[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:20 dist-test-slave-1l3l krb5kdc[24261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:20 dist-test-slave-1l3l krb5kdc[24261](info): set up 2 sockets
Aug 12 14:27:20 dist-test-slave-1l3l krb5kdc[24261](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 14:27:21.499223 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.469s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:21 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:21.618439 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:40325
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:21.701679 24290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:21.701890 24290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:21.701921 24290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:21.703087 24290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:21.703128 24290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:21.703287 24290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:21.703310 24290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:21.703334 24290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:21.705142 24290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:40325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:21.705502 24290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:21.705732 24290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:21.710747 24308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:21.710777 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:21.710816 24290 server_base.cc:1034] running on GCE node
W20260812 14:27:21.710754 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:21.711207 24290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:21.711668 24290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:21.712831 24290 hybrid_clock.cc:648] HybridClock initialized: now 1786544841712811 us; error 70 us; skew 500 ppm
Aug 12 14:27:21 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544841, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:21.715315 24290 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:21.715813 24290 webserver.cc:460] Webserver started at http://127.23.173.254:34503/ using document root <none> and password file <none>
I20260812 14:27:21.716032 24290 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:21.716084 24290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:21.716241 24290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:21.717324 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "0930d119c4c042248b7b43b3f6f6b1e3"
format_stamp: "Formatted at 2026-08-12 14:27:21 on dist-test-slave-1l3l"
server_key: "03e22a6cf0425c49fc4b6760c4a7c3b3"
server_key_iv: "1d51ed2fdd917be9397c2f9f25ed217f"
server_key_version: "encryptionkey@0"
I20260812 14:27:21.717712 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "0930d119c4c042248b7b43b3f6f6b1e3"
format_stamp: "Formatted at 2026-08-12 14:27:21 on dist-test-slave-1l3l"
server_key: "03e22a6cf0425c49fc4b6760c4a7c3b3"
server_key_iv: "1d51ed2fdd917be9397c2f9f25ed217f"
server_key_version: "encryptionkey@0"
I20260812 14:27:21.719223 24290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:21.719918 24316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:21.720100 24290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:21.720185 24290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "0930d119c4c042248b7b43b3f6f6b1e3"
format_stamp: "Formatted at 2026-08-12 14:27:21 on dist-test-slave-1l3l"
server_key: "03e22a6cf0425c49fc4b6760c4a7c3b3"
server_key_iv: "1d51ed2fdd917be9397c2f9f25ed217f"
server_key_version: "encryptionkey@0"
I20260812 14:27:21.720245 24290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:21.735807 24290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:21.737780 24290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:21.737911 24290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:21.742249 24290 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:40325
I20260812 14:27:21.742281 24407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:40325 every 8 connection(s)
I20260812 14:27:21.742676 24290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:21.743113 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24290
I20260812 14:27:21.743230 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:21.743899 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:21.744094 24247 external_mini_cluster.cc:1428] Setting key 29c80046da687663d6614d4aee8de999
I20260812 14:27:21.746886 24408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3: Bootstrap starting.
I20260812 14:27:21.747548 24408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:21.747915 24408 log.cc:824] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:21.748611 24408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3: No bootstrap required, opened a new log
I20260812 14:27:21.751209 24408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40325 } }
I20260812 14:27:21.751336 24408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:21.751359 24408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0930d119c4c042248b7b43b3f6f6b1e3, State: Initialized, Role: FOLLOWER
I20260812 14:27:21.751466 24408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40325 } }
I20260812 14:27:21.751569 24408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:21.751598 24408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:21.751645 24408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:21.752308 24408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40325 } }
I20260812 14:27:21.752420 24408 leader_election.cc:302] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0930d119c4c042248b7b43b3f6f6b1e3; no voters:
I20260812 14:27:21.752574 24408 leader_election.cc:288] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:21.752607 24414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:21.752776 24414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0930d119c4c042248b7b43b3f6f6b1e3, State: Running, Role: LEADER
I20260812 14:27:21.752846 24408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:21.752903 24414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40325 } }
I20260812 14:27:21.753211 24414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0930d119c4c042248b7b43b3f6f6b1e3. Latest consensus state: current_term: 1 leader_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40325 } } }
I20260812 14:27:21.753288 24414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:21.753611 24419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:21.754209 24415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0930d119c4c042248b7b43b3f6f6b1e3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40325 } } }
I20260812 14:27:21.754313 24415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:21.754988 24419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:21.756861 24419 catalog_manager.cc:1347] Generated new cluster ID: d034e16223c54b90b1509a81d68158e0
I20260812 14:27:21.756923 24419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:21 dist-test-slave-1l3l krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:21.761284 24419 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:21.761979 24419 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:21.762352 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:21.745213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.23.173.254:40325)
0812 14:27:21.745430 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:21.745441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:21.745803 (+ 362us) server_negotiation.cc:407] Connection header received
0812 14:27:21.748415 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:21.748421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:21.748465 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:21.748594 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:21.750731 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:21.751765 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:21.753188 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:21.753784 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:21.753899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:21.758178 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:21.758182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:21.758182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:21.758193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:21.760187 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:21.760683 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:21.760686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:21.760686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:21.760774 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:21.761001 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:21.761002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:21.761003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:21.761263 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:21.761414 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:21.761811 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 14:27:21.761952 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260812 14:27:21.769174 24419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0930d119c4c042248b7b43b3f6f6b1e3: Generated new TSK 0
I20260812 14:27:21.769341 24419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:21.798271 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:40325
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:21.908284 24446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:21.908478 24446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:21.908505 24446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:21.909638 24446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:21.909682 24446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:21.909708 24446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:21.911141 24446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:40325
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:21.911357 24446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:21.911576 24446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:21.916510 24451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:21.916532 24455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:21.916534 24453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:21.916711 24446 server_base.cc:1034] running on GCE node
I20260812 14:27:21.916872 24446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:21.917088 24446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:21.918228 24446 hybrid_clock.cc:648] HybridClock initialized: now 1786544841918210 us; error 30 us; skew 500 ppm
Aug 12 14:27:21 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544841, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:21.920683 24446 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:21.921140 24446 webserver.cc:460] Webserver started at http://127.23.173.193:35223/ using document root <none> and password file <none>
I20260812 14:27:21.921358 24446 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:21.921406 24446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:21.921535 24446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:21.922557 24446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "c32d595adf3d4a109c4403daa196514a"
format_stamp: "Formatted at 2026-08-12 14:27:21 on dist-test-slave-1l3l"
server_key: "f22cc883ba0faaca7a8986e07f11a125"
server_key_iv: "123dd2d5b509398f7fb89940c6a2c612"
server_key_version: "encryptionkey@0"
I20260812 14:27:21.922915 24446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "c32d595adf3d4a109c4403daa196514a"
format_stamp: "Formatted at 2026-08-12 14:27:21 on dist-test-slave-1l3l"
server_key: "f22cc883ba0faaca7a8986e07f11a125"
server_key_iv: "123dd2d5b509398f7fb89940c6a2c612"
server_key_version: "encryptionkey@0"
I20260812 14:27:21.926309 24446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 14:27:21.927090 24471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:21.927237 24446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:21.927323 24446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "c32d595adf3d4a109c4403daa196514a"
format_stamp: "Formatted at 2026-08-12 14:27:21 on dist-test-slave-1l3l"
server_key: "f22cc883ba0faaca7a8986e07f11a125"
server_key_iv: "123dd2d5b509398f7fb89940c6a2c612"
server_key_version: "encryptionkey@0"
I20260812 14:27:21.927388 24446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:21.944471 24446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:21.946318 24446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:21.946476 24446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:21.946703 24446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:21.947005 24446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:21.947041 24446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:21.947069 24446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:21.947086 24446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:21.952855 24446 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:40065
I20260812 14:27:21.952921 24670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:40065 every 8 connection(s)
I20260812 14:27:21.953228 24446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:21.954540 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24446
I20260812 14:27:21.954643 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:21.954790 24247 external_mini_cluster.cc:1428] Setting key d806e2a9902580e050a3acca553b8b0f
Aug 12 14:27:21 dist-test-slave-1l3l krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544841, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:21.963272 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:21.954542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:42759 (local address 127.23.173.254:40325)
0812 14:27:21.954600 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:21.954602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:21.954613 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:21.955173 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:21.955174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:21.955187 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:21.955216 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:21.956558 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:21.957059 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:21.958051 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:21.958273 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:21.958329 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:21.960506 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:21.960507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:21.960508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:21.960516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:21.961998 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:21.962371 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:21.962372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:21.962373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:21.962439 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:21.962642 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:21.962643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:21.962643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:21.962774 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:21.962844 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:21.963092 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 14:27:21.963181 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:27:21.963435 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:21.954106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:40325 (local address 127.23.173.193:42759)
0812 14:27:21.954290 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:21.954345 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:21.955051 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:21.955415 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:21.955419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:21.956262 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:21.956463 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:21.956468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:21.957145 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:21.957146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:21.957968 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:21.957971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:21.958344 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:21.958346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:21.958479 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:21.958965 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:21.958973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:21.960410 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:21.962090 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:21.962096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:21.962098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:21.962308 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:21.962501 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:21.962501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:21.962502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:21.962591 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:21.962842 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:21.962847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:21.962967 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 14:27:21.963051 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:27:21.963175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260812 14:27:21.964015 24672 heartbeater.cc:347] Connected to a master server at 127.23.173.254:40325
I20260812 14:27:21.964107 24672 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:21.964272 24672 heartbeater.cc:511] Master 127.23.173.254:40325 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
I20260812 14:27:21.965183 24353 ts_manager.cc:194] Registered new tserver with Master: c32d595adf3d4a109c4403daa196514a (127.23.173.193:40065)
I20260812 14:27:21.966013 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:42759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:21.980212 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:40325
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:22.060245 24682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:22.060433 24682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:22.060456 24682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:22.061563 24682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:22.061605 24682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:22.061630 24682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:22.063073 24682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:40325
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:22.063288 24682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:22.063516 24682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:22.068174 24687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:22.068207 24690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:22.068225 24695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:22.068415 24682 server_base.cc:1034] running on GCE node
I20260812 14:27:22.068619 24682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:22.068862 24682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:22.069994 24682 hybrid_clock.cc:648] HybridClock initialized: now 1786544842069981 us; error 33 us; skew 500 ppm
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:22.072366 24682 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:22.072743 24682 webserver.cc:460] Webserver started at http://127.23.173.194:39591/ using document root <none> and password file <none>
I20260812 14:27:22.072934 24682 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:22.072973 24682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:22.073065 24682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:22.074000 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "00be3b62e3c34ba58050d2ca2f3f8dd6"
format_stamp: "Formatted at 2026-08-12 14:27:22 on dist-test-slave-1l3l"
server_key: "fc8c8991f46f250f5a81cc3343854af4"
server_key_iv: "7ea255868fa670b7cb2063759fa75e2b"
server_key_version: "encryptionkey@0"
I20260812 14:27:22.074296 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "00be3b62e3c34ba58050d2ca2f3f8dd6"
format_stamp: "Formatted at 2026-08-12 14:27:22 on dist-test-slave-1l3l"
server_key: "fc8c8991f46f250f5a81cc3343854af4"
server_key_iv: "7ea255868fa670b7cb2063759fa75e2b"
server_key_version: "encryptionkey@0"
I20260812 14:27:22.075510 24682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:27:22.076210 24709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.076346 24682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.076428 24682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "00be3b62e3c34ba58050d2ca2f3f8dd6"
format_stamp: "Formatted at 2026-08-12 14:27:22 on dist-test-slave-1l3l"
server_key: "fc8c8991f46f250f5a81cc3343854af4"
server_key_iv: "7ea255868fa670b7cb2063759fa75e2b"
server_key_version: "encryptionkey@0"
I20260812 14:27:22.076488 24682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:22.100838 24682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:22.102559 24682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:22.102705 24682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:22.102923 24682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:22.103210 24682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:22.103245 24682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.103274 24682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:22.103297 24682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.109037 24682 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:42193
I20260812 14:27:22.109114 24907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:42193 every 8 connection(s)
I20260812 14:27:22.109385 24682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:22.115003 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24682
I20260812 14:27:22.115082 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:22.115231 24247 external_mini_cluster.cc:1428] Setting key d6a6a3bbde450f2570abe61969af60de
I20260812 14:27:22.118497 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:22.110162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:50017 (local address 127.23.173.254:40325)
0812 14:27:22.110249 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:22.110251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:22.110422 (+ 171us) server_negotiation.cc:407] Connection header received
0812 14:27:22.110880 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:22.110880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:22.110891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:22.110940 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:22.111851 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.112334 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.113358 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.113547 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.113591 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:22.115692 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:22.115694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:22.115694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:22.115724 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:22.117103 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.117580 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.117581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.117582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.117638 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.117881 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.117881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.117882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.117984 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:22.118035 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:22.118336 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 14:27:22.118434 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 14:27:22.118721 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:22.110136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:40325 (local address 127.23.173.194:50017)
0812 14:27:22.110298 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:22.110335 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:22.110799 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:22.110994 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:22.110997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:22.111550 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:22.111781 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:22.111787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.112401 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.112402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:22.113289 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:22.113296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.113603 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.113604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:22.113708 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:22.114144 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:22.114152 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:22.115623 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:22.117190 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:22.117197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:22.117199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:22.117489 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:22.117692 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:22.117693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:22.117693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:22.117825 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:22.118039 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:22.118043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:22.118198 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 14:27:22.118309 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 14:27:22.118489 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260812 14:27:22.119204 24908 heartbeater.cc:347] Connected to a master server at 127.23.173.254:40325
I20260812 14:27:22.119293 24908 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:22.119485 24908 heartbeater.cc:511] Master 127.23.173.254:40325 requested a full tablet report, sending...
I20260812 14:27:22.119872 24353 ts_manager.cc:194] Registered new tserver with Master: 00be3b62e3c34ba58050d2ca2f3f8dd6 (127.23.173.194:42193)
I20260812 14:27:22.120280 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:50017
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:22.142803 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:40325
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:22.224056 24918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:22.224259 24918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:22.224291 24918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:22.225551 24918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:22.225605 24918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:22.225646 24918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:22.227237 24918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:40325
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:22.227496 24918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:22.227746 24918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:22.232896 24926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:22.233049 24918 server_base.cc:1034] running on GCE node
W20260812 14:27:22.232889 24927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:22.232889 24930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:22.233377 24918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:22.233636 24918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:22.234783 24918 hybrid_clock.cc:648] HybridClock initialized: now 1786544842234768 us; error 28 us; skew 500 ppm
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:22.237345 24918 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:22.237809 24918 webserver.cc:460] Webserver started at http://127.23.173.195:34765/ using document root <none> and password file <none>
I20260812 14:27:22.238041 24918 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:22.238092 24918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:22.238220 24918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:22.239292 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "8faec7b8958546ef96aa7fc54fce2fcc"
format_stamp: "Formatted at 2026-08-12 14:27:22 on dist-test-slave-1l3l"
server_key: "f96cdcc01b55a7ba7cc046e5327a5eea"
server_key_iv: "5413b41f438b36ec8f4857ff27f93bf8"
server_key_version: "encryptionkey@0"
I20260812 14:27:22.239640 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "8faec7b8958546ef96aa7fc54fce2fcc"
format_stamp: "Formatted at 2026-08-12 14:27:22 on dist-test-slave-1l3l"
server_key: "f96cdcc01b55a7ba7cc046e5327a5eea"
server_key_iv: "5413b41f438b36ec8f4857ff27f93bf8"
server_key_version: "encryptionkey@0"
I20260812 14:27:22.241050 24918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:22.241783 24940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.241933 24918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:22.242017 24918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "8faec7b8958546ef96aa7fc54fce2fcc"
format_stamp: "Formatted at 2026-08-12 14:27:22 on dist-test-slave-1l3l"
server_key: "f96cdcc01b55a7ba7cc046e5327a5eea"
server_key_iv: "5413b41f438b36ec8f4857ff27f93bf8"
server_key_version: "encryptionkey@0"
I20260812 14:27:22.242089 24918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:22.253082 24918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:22.255311 24918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:22.255506 24918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:22.255750 24918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:22.256088 24918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:22.256125 24918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.256155 24918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:22.256172 24918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:22.262351 24918 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:39343
I20260812 14:27:22.262406 25133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:39343 every 8 connection(s)
I20260812 14:27:22.262738 24918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:22.268102 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24918
I20260812 14:27:22.268193 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:22.268365 24247 external_mini_cluster.cc:1428] Setting key d346f6ea317f8d9056ea6ccf185074c0
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:22.272353 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:22.263502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60933 (local address 127.23.173.254:40325)
0812 14:27:22.263562 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:22.263564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:22.263780 (+ 216us) server_negotiation.cc:407] Connection header received
0812 14:27:22.264334 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:22.264335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:22.264346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:22.264385 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:22.265418 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.265910 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.266893 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.267090 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.267147 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:22.269815 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:22.269816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:22.269817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:22.269825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:22.271027 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.271455 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.271455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.271456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.271509 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.271743 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.271744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.271744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.271863 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:22.271917 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:22.272201 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 14:27:22.272281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:27:22.272493 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:22.263418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:40325 (local address 127.23.173.195:60933)
0812 14:27:22.263632 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:22.263668 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:22.264235 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:22.264468 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:22.264472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:22.265136 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:22.265342 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:22.265348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.265999 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.266000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:22.266805 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:22.266808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.267159 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.267160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:22.267274 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:22.267803 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:22.267812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:22.269728 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:22.271113 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:22.271120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:22.271122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:22.271379 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:22.271562 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:22.271563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:22.271563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:22.271667 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:22.271918 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:22.271923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:22.272054 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 14:27:22.272142 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:27:22.272269 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260812 14:27:22.272974 25134 heartbeater.cc:347] Connected to a master server at 127.23.173.254:40325
I20260812 14:27:22.273080 25134 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:22.273252 25134 heartbeater.cc:511] Master 127.23.173.254:40325 requested a full tablet report, sending...
I20260812 14:27:22.273599 24353 ts_manager.cc:194] Registered new tserver with Master: 8faec7b8958546ef96aa7fc54fce2fcc (127.23.173.195:39343)
I20260812 14:27:22.273947 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:60933
I20260812 14:27:22.280728 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:22.303217 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:22.296433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.23.173.193:40065)
0812 14:27:22.296601 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:22.296606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:22.296617 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:22.296644 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:22.296646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:22.296660 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:22.296734 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:22.297233 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.297935 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.298590 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.298896 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.298945 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:22.300965 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:22.300974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:22.300975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:22.300982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:22.302305 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.302593 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.302595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.302595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.302657 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.302798 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.302799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.302799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.302915 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:22.303017 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:22.303067 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:27:22.303138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
W20260812 14:27:22.303681 24524 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45122
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:22.325425 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:22.319086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.23.173.193:40065)
0812 14:27:22.319138 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:22.319139 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:22.319223 (+ 84us) server_negotiation.cc:407] Connection header received
0812 14:27:22.319270 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:22.319271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:22.319279 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:22.319331 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:22.319873 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.320346 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.320999 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:22.321192 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:22.321234 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:22.323280 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:22.323281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:22.323282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:22.323290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:22.324442 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.324752 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.324753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.324753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.324802 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:22.324965 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:22.324966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:22.324966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:22.325059 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:22.325171 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:22.325254 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:27:22.325333 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:27:22.325961 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24446
I20260812 14:27:22.335521 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24682
I20260812 14:27:22.354239 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24918
I20260812 14:27:22.364811 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24290
2026-08-12T14:27:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2364 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[25158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[25158](info): set up 2 sockets
Aug 12 14:27:22 dist-test-slave-1l3l krb5kdc[25158](info): commencing operation
krb5kdc: starting...
W20260812 14:27:23.535565 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:23 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:23.651842 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:36283
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:23.731241 25177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:23.731431 25177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:23.731460 25177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:23.732591 25177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:23.732635 25177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:23.732651 25177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:23.732667 25177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:23.732681 25177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:23.734123 25177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:23.734354 25177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:23.734552 25177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:23.739259 25184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:23.739259 25187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:23.739557 25177 server_base.cc:1034] running on GCE node
W20260812 14:27:23.739274 25185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:23.739833 25177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:23.740128 25177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:23.741266 25177 hybrid_clock.cc:648] HybridClock initialized: now 1786544843741248 us; error 35 us; skew 500 ppm
Aug 12 14:27:23 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544843, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:23.749655 25177 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:23.750135 25177 webserver.cc:460] Webserver started at http://127.23.173.254:40377/ using document root <none> and password file <none>
I20260812 14:27:23.750351 25177 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:23.750396 25177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:23.750519 25177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:23.751600 25177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "c97be206ba13480eb0f1bba264387a57"
format_stamp: "Formatted at 2026-08-12 14:27:23 on dist-test-slave-1l3l"
server_key: "c817c510abed8b83f1e650cf097a9f31"
server_key_iv: "235325146c01de47637ee4b92b40ed6b"
server_key_version: "encryptionkey@0"
I20260812 14:27:23.751998 25177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "c97be206ba13480eb0f1bba264387a57"
format_stamp: "Formatted at 2026-08-12 14:27:23 on dist-test-slave-1l3l"
server_key: "c817c510abed8b83f1e650cf097a9f31"
server_key_iv: "235325146c01de47637ee4b92b40ed6b"
server_key_version: "encryptionkey@0"
I20260812 14:27:23.753335 25177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:23.754191 25203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:23.754356 25177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:23.754444 25177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "c97be206ba13480eb0f1bba264387a57"
format_stamp: "Formatted at 2026-08-12 14:27:23 on dist-test-slave-1l3l"
server_key: "c817c510abed8b83f1e650cf097a9f31"
server_key_iv: "235325146c01de47637ee4b92b40ed6b"
server_key_version: "encryptionkey@0"
I20260812 14:27:23.754532 25177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:23.772135 25177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:23.774050 25177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:23.774174 25177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:23.778152 25177 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:36283
I20260812 14:27:23.778203 25302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:36283 every 8 connection(s)
I20260812 14:27:23.778517 25177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:23.779364 25303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:23.781718 25303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57: Bootstrap starting.
I20260812 14:27:23.782270 25303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:23.782505 25303 log.cc:824] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:23.783200 25303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57: No bootstrap required, opened a new log
I20260812 14:27:23.784825 25303 raft_consensus.cc:348] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97be206ba13480eb0f1bba264387a57" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36283 } }
I20260812 14:27:23.784973 25303 raft_consensus.cc:374] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:23.784996 25303 raft_consensus.cc:729] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97be206ba13480eb0f1bba264387a57, State: Initialized, Role: FOLLOWER
I20260812 14:27:23.785109 25303 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97be206ba13480eb0f1bba264387a57" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36283 } }
I20260812 14:27:23.785187 25303 raft_consensus.cc:388] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:23.785213 25303 raft_consensus.cc:482] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:23.785245 25303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:23.785717 25303 raft_consensus.cc:504] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97be206ba13480eb0f1bba264387a57" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36283 } }
I20260812 14:27:23.785805 25303 leader_election.cc:302] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97be206ba13480eb0f1bba264387a57; no voters:
I20260812 14:27:23.785933 25303 leader_election.cc:288] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:23.785982 25307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:23.786154 25303 sys_catalog.cc:564] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:23.786140 25307 raft_consensus.cc:686] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [term 1 LEADER]: Becoming Leader. State: Replica: c97be206ba13480eb0f1bba264387a57, State: Running, Role: LEADER
I20260812 14:27:23.786299 25307 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97be206ba13480eb0f1bba264387a57" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36283 } }
I20260812 14:27:23.786334 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25177
I20260812 14:27:23.786405 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:23.786572 24247 external_mini_cluster.cc:1428] Setting key e23def3a81c7a1a9dbcc7ae52350b51b
I20260812 14:27:23.786935 25311 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97be206ba13480eb0f1bba264387a57. Latest consensus state: current_term: 1 leader_uuid: "c97be206ba13480eb0f1bba264387a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97be206ba13480eb0f1bba264387a57" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36283 } } }
I20260812 14:27:23.787017 25311 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:23.787418 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97be206ba13480eb0f1bba264387a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97be206ba13480eb0f1bba264387a57" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36283 } } }
I20260812 14:27:23.787494 25330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:23.787593 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:23.788098 25330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:23.792187 25330 catalog_manager.cc:1347] Generated new cluster ID: f2bf7543870d4d5c89d4286c6457394a
I20260812 14:27:23.792239 25330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:23 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:23.797232 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:23.787233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.23.173.254:36283)
0812 14:27:23.787400 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:23.787409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:23.787443 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:27:23.787562 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:23.787570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:23.787626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:23.787834 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:23.788502 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:23.789756 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:23.790490 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:23.790914 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:23.791011 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:23.793694 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:23.793698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:23.793698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:23.793710 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:23.795593 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:23.796029 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:23.796032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:23.796033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:23.796152 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:23.796355 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:23.796356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:23.796356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:23.796636 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:23.796778 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:23.796891 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 14:27:23.797022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260812 14:27:23.798722 25330 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:23.799399 25330 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:23.807740 25330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c97be206ba13480eb0f1bba264387a57: Generated new TSK 0
I20260812 14:27:23.807965 25330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:23.838303 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:36283
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:23.918941 25346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:23.919144 25346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:23.919176 25346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:23.920356 25346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:23.920397 25346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:23.920431 25346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:23.921936 25346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:36283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:23.922159 25346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:23.922369 25346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:23.927052 25354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:23.927057 25355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:23.927246 25346 server_base.cc:1034] running on GCE node
W20260812 14:27:23.927066 25360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:23.927567 25346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:23.929877 25346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:23.931031 25346 hybrid_clock.cc:648] HybridClock initialized: now 1786544843931014 us; error 35 us; skew 500 ppm
Aug 12 14:27:23 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544843, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:23.933494 25346 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:23.933861 25346 webserver.cc:460] Webserver started at http://127.23.173.193:33547/ using document root <none> and password file <none>
I20260812 14:27:23.934058 25346 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:23.934103 25346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:23.934206 25346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:23.935197 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "09347835033d4124bfb0273c6377fa56"
format_stamp: "Formatted at 2026-08-12 14:27:23 on dist-test-slave-1l3l"
server_key: "18be52e9f6f03ed262ee1c9cb3213809"
server_key_iv: "af6a1e18dddaf3a9de9a7a7a0d721b28"
server_key_version: "encryptionkey@0"
I20260812 14:27:23.935528 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "09347835033d4124bfb0273c6377fa56"
format_stamp: "Formatted at 2026-08-12 14:27:23 on dist-test-slave-1l3l"
server_key: "18be52e9f6f03ed262ee1c9cb3213809"
server_key_iv: "af6a1e18dddaf3a9de9a7a7a0d721b28"
server_key_version: "encryptionkey@0"
I20260812 14:27:23.936795 25346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:23.937469 25370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:23.937614 25346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:23.937687 25346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "09347835033d4124bfb0273c6377fa56"
format_stamp: "Formatted at 2026-08-12 14:27:23 on dist-test-slave-1l3l"
server_key: "18be52e9f6f03ed262ee1c9cb3213809"
server_key_iv: "af6a1e18dddaf3a9de9a7a7a0d721b28"
server_key_version: "encryptionkey@0"
I20260812 14:27:23.937752 25346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:23.952801 25346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:23.954979 25346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:23.955102 25346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:23.955313 25346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:23.955602 25346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:23.955636 25346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:23.955657 25346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:23.955677 25346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:23.961350 25346 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:45033
I20260812 14:27:23.961454 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:45033 every 8 connection(s)
I20260812 14:27:23.961755 25346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:23.963631 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25346
I20260812 14:27:23.963730 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:23.963883 24247 external_mini_cluster.cc:1428] Setting key 329478c3dcda14f848c436b6990b1223
Aug 12 14:27:23 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544843, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:23.972924 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:23.962920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:42129 (local address 127.23.173.254:36283)
0812 14:27:23.962999 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:23.963001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:23.963534 (+ 533us) server_negotiation.cc:407] Connection header received
0812 14:27:23.964785 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:23.964787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:23.964798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:23.964848 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:23.965438 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:23.965994 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:23.967398 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:23.967689 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:23.967788 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:23.969967 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:23.969970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:23.969970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:23.969986 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:23.971580 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:23.972017 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:23.972018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:23.972019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:23.972082 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:23.972311 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:23.972312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:23.972312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:23.972451 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:23.972528 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:23.972744 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 14:27:23.972832 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:27:23.972995 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:23.962874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:36283 (local address 127.23.173.193:42129)
0812 14:27:23.963395 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:23.963443 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:23.964692 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:23.964917 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:23.964921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:23.965166 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:23.965346 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:23.965350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:23.966102 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:23.966103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:23.967302 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:23.967305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:23.967798 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:23.967799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:23.967916 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:23.968363 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:23.968370 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:23.969871 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:23.971666 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:23.971674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:23.971675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:23.971938 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:23.972151 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:23.972151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:23.972152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:23.972251 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:23.972511 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:23.972514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:23.972622 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 14:27:23.972698 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:27:23.972821 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
I20260812 14:27:23.973470 25563 heartbeater.cc:347] Connected to a master server at 127.23.173.254:36283
I20260812 14:27:23.973556 25563 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:23.973727 25563 heartbeater.cc:511] Master 127.23.173.254:36283 requested a full tablet report, sending...
I20260812 14:27:23.974215 25234 ts_manager.cc:194] Registered new tserver with Master: 09347835033d4124bfb0273c6377fa56 (127.23.173.193:45033)
I20260812 14:27:23.974875 25234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:42129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:23.990120 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:36283
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:24.071267 25571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:24.071470 25571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:24.071499 25571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:24.072638 25571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:24.072678 25571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:24.072703 25571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:24.074152 25571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:36283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:24.074370 25571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:24.074594 25571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:24.079531 25578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:24.079522 25580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:24.079576 25571 server_base.cc:1034] running on GCE node
W20260812 14:27:24.079523 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:24.079933 25571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:24.080188 25571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:24.081354 25571 hybrid_clock.cc:648] HybridClock initialized: now 1786544844081338 us; error 30 us; skew 500 ppm
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:24.083958 25571 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:24.084386 25571 webserver.cc:460] Webserver started at http://127.23.173.194:39051/ using document root <none> and password file <none>
I20260812 14:27:24.084626 25571 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:24.084679 25571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:24.084810 25571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:24.085844 25571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "e7fc0f37861147d0bf1e5a215ee78dbb"
format_stamp: "Formatted at 2026-08-12 14:27:24 on dist-test-slave-1l3l"
server_key: "bf0d994b6fca9341061ccab8c01d3e29"
server_key_iv: "70b3b1048c72629e7d6cd39332dc953a"
server_key_version: "encryptionkey@0"
I20260812 14:27:24.086202 25571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "e7fc0f37861147d0bf1e5a215ee78dbb"
format_stamp: "Formatted at 2026-08-12 14:27:24 on dist-test-slave-1l3l"
server_key: "bf0d994b6fca9341061ccab8c01d3e29"
server_key_iv: "70b3b1048c72629e7d6cd39332dc953a"
server_key_version: "encryptionkey@0"
I20260812 14:27:24.087651 25571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:24.088439 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:24.088596 25571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:24.088675 25571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "e7fc0f37861147d0bf1e5a215ee78dbb"
format_stamp: "Formatted at 2026-08-12 14:27:24 on dist-test-slave-1l3l"
server_key: "bf0d994b6fca9341061ccab8c01d3e29"
server_key_iv: "70b3b1048c72629e7d6cd39332dc953a"
server_key_version: "encryptionkey@0"
I20260812 14:27:24.088749 25571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:24.100695 25571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:24.102504 25571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:24.102632 25571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:24.102831 25571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:24.103129 25571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:24.103169 25571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:24.103191 25571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:24.103214 25571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:24.108855 25571 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:40531
I20260812 14:27:24.108917 25797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:40531 every 8 connection(s)
I20260812 14:27:24.109196 25571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:24.115054 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25571
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:24.115129 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:24.115280 24247 external_mini_cluster.cc:1428] Setting key 9527b36145e0b96b2c36e092ea371403
I20260812 14:27:24.118379 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.109916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:53755 (local address 127.23.173.254:36283)
0812 14:27:24.109980 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.109982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.110160 (+ 178us) server_negotiation.cc:407] Connection header received
0812 14:27:24.111048 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.111049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.111059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.111133 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:24.111825 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.112326 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.113293 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.113481 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.113529 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.115623 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:24.115624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:24.115625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:24.115633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:24.117044 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.117413 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.117414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.117414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.117503 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.117771 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.117771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.117772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.117881 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:24.117934 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.118237 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.118315 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:27:24.118606 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.109871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:36283 (local address 127.23.173.194:53755)
0812 14:27:24.110030 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.110074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.110963 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.111217 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.111221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.111513 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:24.111749 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.111754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.112393 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.112393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.113229 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.113231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.113547 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.113548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.113650 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.114101 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:24.114110 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:24.115518 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:24.117117 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:24.117124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:24.117126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:24.117343 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:24.117564 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:24.117564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:24.117565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:24.117703 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:24.117935 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:24.117941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:24.118109 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.118218 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.118386 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260812 14:27:24.119057 25801 heartbeater.cc:347] Connected to a master server at 127.23.173.254:36283
I20260812 14:27:24.119153 25801 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:24.119328 25801 heartbeater.cc:511] Master 127.23.173.254:36283 requested a full tablet report, sending...
I20260812 14:27:24.119729 25234 ts_manager.cc:194] Registered new tserver with Master: e7fc0f37861147d0bf1e5a215ee78dbb (127.23.173.194:40531)
I20260812 14:27:24.120105 25234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:53755
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:24.143565 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:36283
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:24.226477 25811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:24.226673 25811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:24.226696 25811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:24.227804 25811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:24.227839 25811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:24.227864 25811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:24.229252 25811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:36283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:24.229441 25811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:24.229626 25811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:24.234637 25820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:24.234616 25818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:24.234643 25811 server_base.cc:1034] running on GCE node
W20260812 14:27:24.234607 25824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:24.235015 25811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:24.235224 25811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:24.236361 25811 hybrid_clock.cc:648] HybridClock initialized: now 1786544844236341 us; error 35 us; skew 500 ppm
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:24.238844 25811 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:24.239281 25811 webserver.cc:460] Webserver started at http://127.23.173.195:36285/ using document root <none> and password file <none>
I20260812 14:27:24.239481 25811 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:24.239532 25811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:24.239658 25811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:24.240756 25811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "6e19ab01a2eb478a9cfc1eb70aa02793"
format_stamp: "Formatted at 2026-08-12 14:27:24 on dist-test-slave-1l3l"
server_key: "6143838967345970d69b661530fe9757"
server_key_iv: "16e762394c9e702ae0cd3a3cec20542b"
server_key_version: "encryptionkey@0"
I20260812 14:27:24.241082 25811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "6e19ab01a2eb478a9cfc1eb70aa02793"
format_stamp: "Formatted at 2026-08-12 14:27:24 on dist-test-slave-1l3l"
server_key: "6143838967345970d69b661530fe9757"
server_key_iv: "16e762394c9e702ae0cd3a3cec20542b"
server_key_version: "encryptionkey@0"
I20260812 14:27:24.242322 25811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:24.243001 25832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:24.243165 25811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:24.243244 25811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "6e19ab01a2eb478a9cfc1eb70aa02793"
format_stamp: "Formatted at 2026-08-12 14:27:24 on dist-test-slave-1l3l"
server_key: "6143838967345970d69b661530fe9757"
server_key_iv: "16e762394c9e702ae0cd3a3cec20542b"
server_key_version: "encryptionkey@0"
I20260812 14:27:24.243297 25811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:24.251678 25811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:24.253451 25811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:24.253571 25811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:24.253752 25811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:24.254016 25811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:24.254065 25811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:24.254088 25811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:24.254102 25811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:24.259796 25811 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:45355
I20260812 14:27:24.259866 26021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:45355 every 8 connection(s)
I20260812 14:27:24.260154 25811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:24.269070 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.260835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:53503 (local address 127.23.173.254:36283)
0812 14:27:24.260897 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.260899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.261098 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:27:24.262032 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.262033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.262047 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.262100 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:24.262714 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.263211 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.264299 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.264495 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.264552 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.266547 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:24.266548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:24.266548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:24.266555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:24.267809 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.268181 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.268182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.268183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.268236 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.268450 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.268451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.268451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.268574 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:24.268632 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.268892 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.268979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:27:24.269240 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.260804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:36283 (local address 127.23.173.195:53503)
0812 14:27:24.260976 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.261021 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.261924 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.262172 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.262175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.262434 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:24.262624 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.262629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.263295 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.263296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.264213 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.264217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.264566 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.264567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.264683 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.265153 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:24.265161 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:24.266450 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:24.267874 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:24.267882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:24.267884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:24.268113 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:24.268288 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:24.268288 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:24.268289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:24.268394 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:24.268629 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:24.268636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:24.268771 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.268861 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.269028 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260812 14:27:24.269423 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25811
I20260812 14:27:24.269529 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:24.269711 24247 external_mini_cluster.cc:1428] Setting key 4b69a9a34d1e735afcb14c3f1ad4bd7d
I20260812 14:27:24.269707 26024 heartbeater.cc:347] Connected to a master server at 127.23.173.254:36283
I20260812 14:27:24.269797 26024 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:24.269969 26024 heartbeater.cc:511] Master 127.23.173.254:36283 requested a full tablet report, sending...
I20260812 14:27:24.270324 25234 ts_manager.cc:194] Registered new tserver with Master: 6e19ab01a2eb478a9cfc1eb70aa02793 (127.23.173.195:45355)
I20260812 14:27:24.270594 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:27:24.270779 25234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:53503
I20260812 14:27:24.277875 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.272585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.23.173.254:36283)
0812 14:27:24.272622 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.272624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.272753 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:27:24.272792 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.272792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.272800 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.272844 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:24.273317 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.273765 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.274461 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.274631 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.274672 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.275694 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:24.275695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:24.275695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:24.275727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:24.276952 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.277236 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.277236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.277237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.277281 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.277400 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.277401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.277401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.277526 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:24.277594 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.277759 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.277815 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:27:24.280596 25234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:24.282316 25234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:24.289492 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.285161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:45355 (local address 127.0.0.1:33474)
0812 14:27:24.285335 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.285374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.285438 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.285849 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.285853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.285855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:24.285929 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.285931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.287406 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.287408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.288311 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.288315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.289176 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.289177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.289253 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.289273 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.289349 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.289402 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
I20260812 14:27:24.289657 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.285398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:40531 (local address 127.0.0.1:48480)
0812 14:27:24.285584 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.285593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.285640 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.285919 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.285920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.285922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:24.285976 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.285978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.287386 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.287386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.288573 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.288575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.289454 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.289455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.289514 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.289519 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.289549 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.289587 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":25,"thread_start_us":37,"threads_started":1}
I20260812 14:27:24.289788 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.285161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:45033 (local address 127.0.0.1:47574)
0812 14:27:24.285326 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.285373 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.285438 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.286022 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.286022 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.286024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:24.286057 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.286059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.287187 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.287188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.288763 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.288765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.289577 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.289578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.289640 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.289642 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.289690 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.289728 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":33,"thread_start_us":60,"threads_started":1}
I20260812 14:27:24.289835 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.285278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.23.173.193:45033)
0812 14:27:24.285775 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.285780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.285790 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:24.285833 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.285835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.285851 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.285929 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:24.286141 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.287092 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.288830 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.289426 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.289489 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.289542 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.289718 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.289764 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":91,"threads_started":1}
I20260812 14:27:24.290047 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.285602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.23.173.194:40531)
0812 14:27:24.285762 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.285766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.285774 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:24.285812 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.285814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.285827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.286061 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:24.286144 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.287220 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.288651 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.289360 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.289814 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.289879 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.289906 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.289953 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260812 14:27:24.289496 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.285487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.23.173.195:45355)
0812 14:27:24.285691 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.285696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.285706 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:24.285746 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.285748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.285765 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.285842 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:24.286006 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.287304 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.288425 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.289094 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.289171 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.289228 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.289372 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.289416 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":116,"threads_started":1}
I20260812 14:27:24.290478 25454 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3d2f6f70e946be8cb119dc023039a1 (DEFAULT_TABLE table=test-table [id=0c05005f94a64e88b5e3357665563064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:24.290874 25454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3d2f6f70e946be8cb119dc023039a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:24.290831 25917 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3d2f6f70e946be8cb119dc023039a1 (DEFAULT_TABLE table=test-table [id=0c05005f94a64e88b5e3357665563064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:24.291224 25917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3d2f6f70e946be8cb119dc023039a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:24.293670 26047 tablet_bootstrap.cc:492] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56: Bootstrap starting.
I20260812 14:27:24.294123 26049 tablet_bootstrap.cc:492] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793: Bootstrap starting.
I20260812 14:27:24.294282 26047 tablet_bootstrap.cc:654] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:24.294327 25675 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3d2f6f70e946be8cb119dc023039a1 (DEFAULT_TABLE table=test-table [id=0c05005f94a64e88b5e3357665563064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:24.294668 25675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3d2f6f70e946be8cb119dc023039a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:24.294718 26047 log.cc:824] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:24.294744 26049 tablet_bootstrap.cc:654] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:24.295281 26049 log.cc:824] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:24.295372 26047 tablet_bootstrap.cc:492] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56: No bootstrap required, opened a new log
I20260812 14:27:24.295418 26047 ts_tablet_manager.cc:1397] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:24.296236 26049 tablet_bootstrap.cc:492] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793: No bootstrap required, opened a new log
I20260812 14:27:24.296311 26049 ts_tablet_manager.cc:1397] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:24.296924 26047 raft_consensus.cc:348] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.297070 26047 raft_consensus.cc:374] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:24.297104 26047 raft_consensus.cc:729] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09347835033d4124bfb0273c6377fa56, State: Initialized, Role: FOLLOWER
I20260812 14:27:24.297237 26047 consensus_queue.cc:260] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.297529 26047 ts_tablet_manager.cc:1428] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:27:24.297658 25563 heartbeater.cc:503] Master 127.23.173.254:36283 was elected leader, sending a full tablet report...
I20260812 14:27:24.297991 26056 tablet_bootstrap.cc:492] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: Bootstrap starting.
I20260812 14:27:24.298422 26049 raft_consensus.cc:348] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.298554 26049 raft_consensus.cc:374] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:24.298576 26049 raft_consensus.cc:729] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e19ab01a2eb478a9cfc1eb70aa02793, State: Initialized, Role: FOLLOWER
I20260812 14:27:24.298686 26049 consensus_queue.cc:260] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.298771 26056 tablet_bootstrap.cc:654] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:24.298964 26049 ts_tablet_manager.cc:1428] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:27:24.299120 26024 heartbeater.cc:503] Master 127.23.173.254:36283 was elected leader, sending a full tablet report...
I20260812 14:27:24.299343 26056 log.cc:824] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:24.300132 26056 tablet_bootstrap.cc:492] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: No bootstrap required, opened a new log
I20260812 14:27:24.300202 26056 ts_tablet_manager.cc:1397] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:24.301486 26056 raft_consensus.cc:348] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.301622 26056 raft_consensus.cc:374] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:24.301643 26056 raft_consensus.cc:729] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fc0f37861147d0bf1e5a215ee78dbb, State: Initialized, Role: FOLLOWER
I20260812 14:27:24.301723 26056 consensus_queue.cc:260] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.301913 26056 ts_tablet_manager.cc:1428] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:24.301962 25801 heartbeater.cc:503] Master 127.23.173.254:36283 was elected leader, sending a full tablet report...
W20260812 14:27:24.359773 25802 tablet.cc:2369] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:27:24.378708 26054 raft_consensus.cc:482] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:24.378866 26054 raft_consensus.cc:504] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.379420 26054 leader_election.cc:288] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7fc0f37861147d0bf1e5a215ee78dbb (127.23.173.194:40531), 6e19ab01a2eb478a9cfc1eb70aa02793 (127.23.173.195:45355)
I20260812 14:27:24.382292 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.379614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:45355 (local address 127.23.173.193:44885)
0812 14:27:24.379682 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.379694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.379761 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.379905 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.379907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.379909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:24.380002 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.380003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.380702 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.380703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.381471 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.381473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.382029 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.382031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.382110 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.382113 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.382159 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.382203 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 14:27:24.382277 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.379685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:44885 (local address 127.23.173.195:45355)
0812 14:27:24.379770 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.379772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.379783 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:24.379813 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.379814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.379824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.379887 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:24.380070 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.380617 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.381562 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.381948 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.382017 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.382027 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.382175 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.382209 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 14:27:24.382295 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.379686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:56677 (local address 127.23.173.194:40531)
0812 14:27:24.379756 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.379758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.379977 (+ 219us) server_negotiation.cc:407] Connection header received
0812 14:27:24.380026 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.380027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.380036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.380082 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:24.380228 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.380688 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.381597 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.381971 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.382046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.382055 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.382194 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.382230 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:27:24.382313 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.379616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:40531 (local address 127.23.173.193:56677)
0812 14:27:24.379906 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:24.379913 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:24.379961 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:24.380093 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:24.380094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:24.380095 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:24.380166 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.380168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.380766 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.380767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.381507 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:24.381509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.382060 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.382061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:24.382130 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.382133 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:24.382178 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:27:24.382217 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":116,"thread_start_us":111,"threads_started":1}
I20260812 14:27:24.382716 25718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3d2f6f70e946be8cb119dc023039a1" candidate_uuid: "09347835033d4124bfb0273c6377fa56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" is_pre_election: true
I20260812 14:27:24.382885 25718 raft_consensus.cc:2454] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09347835033d4124bfb0273c6377fa56 in term 0.
I20260812 14:27:24.383104 25372 leader_election.cc:302] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09347835033d4124bfb0273c6377fa56, e7fc0f37861147d0bf1e5a215ee78dbb; no voters:
I20260812 14:27:24.383252 26054 raft_consensus.cc:2781] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:24.383314 26054 raft_consensus.cc:482] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:24.383337 26054 raft_consensus.cc:3037] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:24.384124 26054 raft_consensus.cc:504] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.384254 26054 leader_election.cc:288] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [CANDIDATE]: Term 1 election: Requested vote from peers e7fc0f37861147d0bf1e5a215ee78dbb (127.23.173.194:40531), 6e19ab01a2eb478a9cfc1eb70aa02793 (127.23.173.195:45355)
I20260812 14:27:24.384452 25718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3d2f6f70e946be8cb119dc023039a1" candidate_uuid: "09347835033d4124bfb0273c6377fa56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb"
I20260812 14:27:24.384433 25941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3d2f6f70e946be8cb119dc023039a1" candidate_uuid: "09347835033d4124bfb0273c6377fa56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793"
I20260812 14:27:24.384539 25718 raft_consensus.cc:3037] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:24.384562 25941 raft_consensus.cc:3037] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:24.385124 25942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3d2f6f70e946be8cb119dc023039a1" candidate_uuid: "09347835033d4124bfb0273c6377fa56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" is_pre_election: true
I20260812 14:27:24.385488 25718 raft_consensus.cc:2454] T 9d3d2f6f70e946be8cb119dc023039a1 P e7fc0f37861147d0bf1e5a215ee78dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09347835033d4124bfb0273c6377fa56 in term 1.
I20260812 14:27:24.385488 25941 raft_consensus.cc:2454] T 9d3d2f6f70e946be8cb119dc023039a1 P 6e19ab01a2eb478a9cfc1eb70aa02793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09347835033d4124bfb0273c6377fa56 in term 1.
I20260812 14:27:24.385646 25372 leader_election.cc:302] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09347835033d4124bfb0273c6377fa56, e7fc0f37861147d0bf1e5a215ee78dbb; no voters:
I20260812 14:27:24.385758 26054 raft_consensus.cc:2781] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:24.385963 26054 raft_consensus.cc:686] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [term 1 LEADER]: Becoming Leader. State: Replica: 09347835033d4124bfb0273c6377fa56, State: Running, Role: LEADER
I20260812 14:27:24.386063 26054 consensus_queue.cc:237] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } }
I20260812 14:27:24.387131 25231 catalog_manager.cc:5495] T 9d3d2f6f70e946be8cb119dc023039a1 P 09347835033d4124bfb0273c6377fa56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09347835033d4124bfb0273c6377fa56 (127.23.173.193). New cstate: current_term: 1 leader_uuid: "09347835033d4124bfb0273c6377fa56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09347835033d4124bfb0273c6377fa56" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7fc0f37861147d0bf1e5a215ee78dbb" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e19ab01a2eb478a9cfc1eb70aa02793" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 45355 } health_report { overall_health: UNKNOWN } } }
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:24.409373 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.401731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.23.173.193:45033)
0812 14:27:24.401821 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.401823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.401890 (+ 67us) server_negotiation.cc:407] Connection header received
0812 14:27:24.401923 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.401928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.401940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.402007 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:24.402558 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.403050 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.403797 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.404001 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.404050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.406538 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:24.406553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:24.406553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:24.406561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:24.408231 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.408584 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.408587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.408587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.408659 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.408873 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.408873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.408873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.409014 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:24.409134 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.409236 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.409302 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:24.433888 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.426785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.23.173.193:45033)
0812 14:27:24.426870 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.426872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.426908 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:27:24.426969 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.426971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.426980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.427037 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:24.427541 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.428071 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.428814 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.429026 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.429081 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.431283 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:24.431285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:24.431285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:24.431292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:24.432600 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.432974 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.432976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.432976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.433036 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.433350 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.433352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.433352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.433468 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:24.433583 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.433735 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.433811 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[25158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:24.459139 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:24.451687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.23.173.193:45033)
0812 14:27:24.451789 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:24.451792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:24.451936 (+ 144us) server_negotiation.cc:407] Connection header received
0812 14:27:24.451969 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:24.451970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:24.451983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:24.452038 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:24.452630 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.453138 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.453872 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:24.454084 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:24.454148 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:24.456732 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:24.456734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:24.456734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:24.456742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:24.458080 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.458407 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.458409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.458409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.458462 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:24.458636 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:24.458637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:24.458637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:24.458748 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:24.458843 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:24.458956 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 14:27:24.459045 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260812 14:27:24.459678 25421 tablet_service.cc:2908] Rejecting scan request for tablet 9d3d2f6f70e946be8cb119dc023039a1: Uninitialized: clean time has not yet been initialized
I20260812 14:27:24.460598 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25346
I20260812 14:27:24.479038 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25571
I20260812 14:27:24.489461 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25811
I20260812 14:27:24.505892 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25177
2026-08-12T14:27:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2144 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[26098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[26098](info): set up 2 sockets
Aug 12 14:27:24 dist-test-slave-1l3l krb5kdc[26098](info): commencing operation
krb5kdc: starting...
W20260812 14:27:25.692342 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:25 dist-test-slave-1l3l krb5kdc[26098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:25.810629 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:42487
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:25.891901 26120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:25.892115 26120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:25.892148 26120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:25.893333 26120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:25.893388 26120 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:27:25.893404 26120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:25.893422 26120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:25.893436 26120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:25.893453 26120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:25.895026 26120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:42487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:25.895280 26120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:25.895533 26120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:25.900393 26129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:25.900413 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:25.900504 26120 server_base.cc:1034] running on GCE node
W20260812 14:27:25.900655 26135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:25.900913 26120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:25.901244 26120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:25.902405 26120 hybrid_clock.cc:648] HybridClock initialized: now 1786544845902394 us; error 32 us; skew 500 ppm
Aug 12 14:27:25 dist-test-slave-1l3l krb5kdc[26098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544845, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:25.904917 26120 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:25.905333 26120 webserver.cc:460] Webserver started at http://127.23.173.254:46765/ using document root <none> and password file <none>
I20260812 14:27:25.905545 26120 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:25.905586 26120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:25.905678 26120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:25.906736 26120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "a61ba4a5dc4f4ccbac821de379750f1b"
format_stamp: "Formatted at 2026-08-12 14:27:25 on dist-test-slave-1l3l"
server_key: "062ba9ff7543212eababb24c30ba9e76"
server_key_iv: "d5ba3cbbe197d1b8eb0d04eb65733a63"
server_key_version: "encryptionkey@0"
I20260812 14:27:25.907039 26120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "a61ba4a5dc4f4ccbac821de379750f1b"
format_stamp: "Formatted at 2026-08-12 14:27:25 on dist-test-slave-1l3l"
server_key: "062ba9ff7543212eababb24c30ba9e76"
server_key_iv: "d5ba3cbbe197d1b8eb0d04eb65733a63"
server_key_version: "encryptionkey@0"
I20260812 14:27:25.908407 26120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:25.909170 26145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:25.909374 26120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:25.909487 26120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "a61ba4a5dc4f4ccbac821de379750f1b"
format_stamp: "Formatted at 2026-08-12 14:27:25 on dist-test-slave-1l3l"
server_key: "062ba9ff7543212eababb24c30ba9e76"
server_key_iv: "d5ba3cbbe197d1b8eb0d04eb65733a63"
server_key_version: "encryptionkey@0"
I20260812 14:27:25.909547 26120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:25.924733 26120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:25.926893 26120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:25.927016 26120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:25.931761 26120 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:42487
I20260812 14:27:25.931820 26245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:42487 every 8 connection(s)
I20260812 14:27:25.932181 26120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:25.934551 26247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:25.935578 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26120
I20260812 14:27:25.935684 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:25.935868 24247 external_mini_cluster.cc:1428] Setting key 2c0183d55f690b04818198661a90b45c
I20260812 14:27:25.937515 26247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: Bootstrap starting.
I20260812 14:27:25.938299 26247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:25.938609 26247 log.cc:824] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:25.939325 26247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: No bootstrap required, opened a new log
I20260812 14:27:25.941776 26247 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42487 } }
I20260812 14:27:25.941964 26247 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:25.942027 26247 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61ba4a5dc4f4ccbac821de379750f1b, State: Initialized, Role: FOLLOWER
I20260812 14:27:25.942268 26247 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42487 } }
I20260812 14:27:25.942351 26247 raft_consensus.cc:388] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:25.942387 26247 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:25.942437 26247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:27:25 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:25.943010 26247 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42487 } }
I20260812 14:27:25.943100 26247 leader_election.cc:302] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a61ba4a5dc4f4ccbac821de379750f1b; no voters:
I20260812 14:27:25.943293 26247 leader_election.cc:288] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:25.943347 26256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:25.943517 26256 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [term 1 LEADER]: Becoming Leader. State: Replica: a61ba4a5dc4f4ccbac821de379750f1b, State: Running, Role: LEADER
I20260812 14:27:25.943568 26247 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:25.943616 26256 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42487 } }
I20260812 14:27:25.944028 26258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42487 } } }
I20260812 14:27:25.944092 26258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:25.943962 26256 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61ba4a5dc4f4ccbac821de379750f1b. Latest consensus state: current_term: 1 leader_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61ba4a5dc4f4ccbac821de379750f1b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42487 } } }
I20260812 14:27:25.944332 26256 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:25.944682 26268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:25.946022 26268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:25.947170 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:25.936431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.23.173.254:42487)
0812 14:27:25.936637 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:25.936646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:25.936752 (+ 106us) server_negotiation.cc:407] Connection header received
0812 14:27:25.937352 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:25.937360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:25.937422 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:25.937584 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:25.938332 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:25.939443 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:25.940470 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:25.940906 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:25.940981 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:25.943732 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:25.943737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:25.943737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:25.943751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:25.945569 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:25.945910 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:25.945913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:25.945914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:25.946024 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:25.946248 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:25.946249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:25.946250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:25.946536 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:25.946690 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:25.946799 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 14:27:25.946948 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260812 14:27:25.948166 26268 catalog_manager.cc:1347] Generated new cluster ID: 151ce5a751294d3ea2d782842306a1a3
I20260812 14:27:25.948206 26268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:25.964613 26268 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:25.965154 26268 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:25.969630 26268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: Generated new TSK 0
I20260812 14:27:25.969846 26268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:26.006433 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42487
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:26.125715 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:26.125939 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:26.125968 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:26.127126 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:26.127161 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:26.127180 26287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:27:26.127197 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:26.128739 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:26.128983 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:26.129184 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:26.134572 26300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:26.134667 26287 server_base.cc:1034] running on GCE node
W20260812 14:27:26.134584 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:26.134572 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:26.135118 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:26.135383 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:26.136536 26287 hybrid_clock.cc:648] HybridClock initialized: now 1786544846136527 us; error 39 us; skew 500 ppm
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:26.139182 26287 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:26.139642 26287 webserver.cc:460] Webserver started at http://127.23.173.193:33583/ using document root <none> and password file <none>
I20260812 14:27:26.139910 26287 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:26.140014 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:26.140146 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:26.141242 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "31517d827ca44133b78849003b79d7e9"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "da5e9bf9c9f5468e4cbda5db913bac07"
server_key_iv: "d5241bc1a43bd0bc17b40883dca51383"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.141592 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "31517d827ca44133b78849003b79d7e9"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "da5e9bf9c9f5468e4cbda5db913bac07"
server_key_iv: "d5241bc1a43bd0bc17b40883dca51383"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.143006 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:26.143788 26312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.143939 26287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:26.144038 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "31517d827ca44133b78849003b79d7e9"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "da5e9bf9c9f5468e4cbda5db913bac07"
server_key_iv: "d5241bc1a43bd0bc17b40883dca51383"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.144109 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:26.160338 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:26.162272 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:26.162425 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:26.162973 26287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:26.163014 26287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.163045 26287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:26.163067 26287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:26.170094 26287 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:39007
I20260812 14:27:26.170449 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:26.170423 26508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:39007 every 8 connection(s)
I20260812 14:27:26.172652 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.163561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:42857 (local address 127.23.173.254:42487)
0812 14:27:26.163626 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.163629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.163847 (+ 218us) server_negotiation.cc:407] Connection header received
0812 14:27:26.164363 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.164364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.164379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.164463 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.165150 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.165910 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.166948 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.167162 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.167221 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.169906 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.169908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.169908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.169917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.171333 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.171805 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.171807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.171807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.171867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.172100 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.172100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.172101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.172201 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.172246 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.172502 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.172586 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:27:26.172778 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.163496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.23.173.193:42857)
0812 14:27:26.163684 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.163757 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.164257 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.164532 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.164537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.164838 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.165039 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.165045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.166039 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.166040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.166850 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.166852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.167241 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.167242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.167383 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.167867 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.167876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.169814 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.171423 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.171431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.171433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.171679 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.171926 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.171927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.171927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.172044 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.172243 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.172246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.172388 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.172473 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.172602 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260812 14:27:26.173305 26511 heartbeater.cc:347] Connected to a master server at 127.23.173.254:42487
I20260812 14:27:26.173403 26511 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:26.173444 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26287
I20260812 14:27:26.173540 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:26.173596 26511 heartbeater.cc:511] Master 127.23.173.254:42487 requested a full tablet report, sending...
I20260812 14:27:26.173723 24247 external_mini_cluster.cc:1428] Setting key f074b1d3e3df6ca466978ff1bb11862d
I20260812 14:27:26.174144 26171 ts_manager.cc:194] Registered new tserver with Master: 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007)
I20260812 14:27:26.174908 26171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:42857
I20260812 14:27:26.179762 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.174387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51471 (local address 127.23.173.254:42487)
0812 14:27:26.174422 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.174424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.174549 (+ 125us) server_negotiation.cc:407] Connection header received
0812 14:27:26.174604 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.174604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.174613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.174667 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.174992 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.175515 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.176254 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.176441 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.176497 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.177323 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.177325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.177325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.177331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.178674 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.179030 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.179031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.179031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.179086 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.179256 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.179257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.179257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.179359 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.179416 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.179574 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.179648 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:26.179769 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.174335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.23.173.193:51471)
0812 14:27:26.174477 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.174486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.174548 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.174672 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.174673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.174860 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.174920 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.174922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.175622 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.175623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.176181 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.176183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.176507 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.176507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.176590 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.176984 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.176988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.177243 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.178775 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.178776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.178777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.178945 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.179144 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.179144 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.179145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.179203 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.179416 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.179417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.179463 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.179541 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.179645 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:26.202205 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42487
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:26.287267 26524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:26.287509 26524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:26.287537 26524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:26.288820 26524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:26.288869 26524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:26.288890 26524 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:27:26.288908 26524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:26.290372 26524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:26.290591 26524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:26.290782 26524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:26.295871 26537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:26.295837 26532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:26.296033 26524 server_base.cc:1034] running on GCE node
W20260812 14:27:26.295835 26533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:26.296286 26524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:26.296531 26524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:26.297668 26524 hybrid_clock.cc:648] HybridClock initialized: now 1786544846297650 us; error 37 us; skew 500 ppm
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:26.300138 26524 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:26.300612 26524 webserver.cc:460] Webserver started at http://127.23.173.194:36387/ using document root <none> and password file <none>
I20260812 14:27:26.300822 26524 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:26.300871 26524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:26.300999 26524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:26.302100 26524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "3cebd4169826461aaa88fd73cebb2f36"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "0a0c1ae2b889c05984c39bfb6834e2d6"
server_key_iv: "95647bdabcc9abcd19569bcad6031f88"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.302445 26524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "3cebd4169826461aaa88fd73cebb2f36"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "0a0c1ae2b889c05984c39bfb6834e2d6"
server_key_iv: "95647bdabcc9abcd19569bcad6031f88"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.303689 26524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:27:26.304443 26552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.304600 26524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:26.304684 26524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "3cebd4169826461aaa88fd73cebb2f36"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "0a0c1ae2b889c05984c39bfb6834e2d6"
server_key_iv: "95647bdabcc9abcd19569bcad6031f88"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.304745 26524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:26.325768 26524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:26.327625 26524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:26.327805 26524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:26.328394 26524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:26.328447 26524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.328478 26524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:26.328495 26524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:26.335450 26524 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:43273
I20260812 14:27:26.335520 26739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:43273 every 8 connection(s)
I20260812 14:27:26.335894 26524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:26.338030 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.328933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:45385 (local address 127.23.173.254:42487)
0812 14:27:26.329002 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.329004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.329267 (+ 263us) server_negotiation.cc:407] Connection header received
0812 14:27:26.329798 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.329799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.329813 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.329868 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.330726 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.331491 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.333016 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.333248 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.333309 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.335548 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.335550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.335550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.335559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.336916 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.337277 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.337278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.337278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.337328 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.337521 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.337521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.337522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.337621 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.337668 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.337888 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.337966 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:26.338181 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.328961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.23.173.194:45385)
0812 14:27:26.329148 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.329187 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.329695 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.329956 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.329960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.330320 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.330601 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.330608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.331610 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.331611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.332931 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.332935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.333363 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.333364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.333492 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.333970 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.333982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.335431 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.336996 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.337003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.337004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.337219 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.337365 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.337366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.337366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.337452 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.337666 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.337670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.337791 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.337870 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.337996 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260812 14:27:26.338481 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26524
I20260812 14:27:26.338555 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:26.338676 26740 heartbeater.cc:347] Connected to a master server at 127.23.173.254:42487
I20260812 14:27:26.338742 24247 external_mini_cluster.cc:1428] Setting key 202630c892a3ea73aee9b1d1421ec8fc
I20260812 14:27:26.338759 26740 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:26.338918 26740 heartbeater.cc:511] Master 127.23.173.254:42487 requested a full tablet report, sending...
I20260812 14:27:26.339511 26171 ts_manager.cc:194] Registered new tserver with Master: 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273)
I20260812 14:27:26.340013 26171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:45385
I20260812 14:27:26.346155 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.339871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.23.173.194:46191)
0812 14:27:26.340030 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.340042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.340123 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.340243 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.340244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.340448 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.340519 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.340521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.341304 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.341305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.341814 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.341815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.342306 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.342307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.342380 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.342889 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.342895 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.343198 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.345149 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.345150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.345150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.345324 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.345527 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.345527 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.345527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.345579 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.345901 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.345902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.345948 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.345992 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.346077 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260812 14:27:26.346223 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.339825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:46191 (local address 127.23.173.254:42487)
0812 14:27:26.339883 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.339885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.340097 (+ 212us) server_negotiation.cc:407] Connection header received
0812 14:27:26.340178 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.340179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.340189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.340238 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.340595 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.341199 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.341884 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.342150 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.342219 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.343285 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.343287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.343287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.343295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.345071 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.345394 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.345395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.345395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.345460 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.345630 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.345631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.345631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.345840 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.345910 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.346043 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.346130 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:26.366981 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42487
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:26.450835 26760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:26.451064 26760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:26.451089 26760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:26.452302 26760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:26.452345 26760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:26.452365 26760 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:27:26.452381 26760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:26.453869 26760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:26.454088 26760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:26.454284 26760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:26.459405 26768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:26.459578 26779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:26.459662 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:26.459820 26760 server_base.cc:1034] running on GCE node
I20260812 14:27:26.460067 26760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:26.460336 26760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:26.461504 26760 hybrid_clock.cc:648] HybridClock initialized: now 1786544846461485 us; error 32 us; skew 500 ppm
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:26.464206 26760 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:26.464720 26760 webserver.cc:460] Webserver started at http://127.23.173.195:40129/ using document root <none> and password file <none>
I20260812 14:27:26.464959 26760 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:26.465018 26760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:26.465217 26760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:26.466331 26760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "188fa031a286433da9e688b9992fb685"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "832452942be0e6b54bf1cb41f30171dc"
server_key_iv: "bd7d3556a71fd13fcbf95970f7c97c30"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.466706 26760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "188fa031a286433da9e688b9992fb685"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "832452942be0e6b54bf1cb41f30171dc"
server_key_iv: "bd7d3556a71fd13fcbf95970f7c97c30"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.468297 26760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:26.469063 26792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.469241 26760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:26.469324 26760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "188fa031a286433da9e688b9992fb685"
format_stamp: "Formatted at 2026-08-12 14:27:26 on dist-test-slave-1l3l"
server_key: "832452942be0e6b54bf1cb41f30171dc"
server_key_iv: "bd7d3556a71fd13fcbf95970f7c97c30"
server_key_version: "encryptionkey@0"
I20260812 14:27:26.469408 26760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:26.486325 26760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:26.488667 26760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:26.488793 26760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:26.489352 26760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:26.489403 26760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.489425 26760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:26.489437 26760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:26.496510 26760 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:40651
I20260812 14:27:26.496639 26987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:40651 every 8 connection(s)
I20260812 14:27:26.496927 26760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:26.498955 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.489882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:56639 (local address 127.23.173.254:42487)
0812 14:27:26.489950 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.489952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.490142 (+ 190us) server_negotiation.cc:407] Connection header received
0812 14:27:26.490653 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.490654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.490668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.490740 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.491435 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.492069 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.493247 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.493446 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.493503 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.496170 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.496172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.496172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.496181 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.497688 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.498060 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.498061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.498061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.498121 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.498330 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.498331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.498331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.498453 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.498513 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.498779 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.498864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:27:26.499118 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.489833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.23.173.195:56639)
0812 14:27:26.490006 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.490052 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.490558 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.490840 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.490844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.491132 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.491337 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.491342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.492158 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.492159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.493164 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.493167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.493532 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.493533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.493660 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.494162 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.494170 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.496071 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.497764 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.497770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.497773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.497989 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.498175 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.498176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.498176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.498275 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.498518 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.498525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.498656 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.498745 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.498907 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260812 14:27:26.499596 26988 heartbeater.cc:347] Connected to a master server at 127.23.173.254:42487
I20260812 14:27:26.499675 26988 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:26.499866 26988 heartbeater.cc:511] Master 127.23.173.254:42487 requested a full tablet report, sending...
I20260812 14:27:26.500240 26171 ts_manager.cc:194] Registered new tserver with Master: 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651)
I20260812 14:27:26.500684 26171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:56639
I20260812 14:27:26.502943 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26760
I20260812 14:27:26.503037 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:26.503213 24247 external_mini_cluster.cc:1428] Setting key a90e78be01cacc9f61dbe16bd92b5bf6
I20260812 14:27:26.504259 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:27:26.505712 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.500736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:52499 (local address 127.23.173.254:42487)
0812 14:27:26.500822 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.500824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.500879 (+ 55us) server_negotiation.cc:407] Connection header received
0812 14:27:26.500959 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.500960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.500968 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.501029 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.501320 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.501826 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.502448 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.502619 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.502662 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.503501 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.503502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.503502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.503509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.504738 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.505034 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.505035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.505035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.505090 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.505249 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.505250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.505250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.505352 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.505406 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.505547 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.505620 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 14:27:26.505723 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.500672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.23.173.195:52499)
0812 14:27:26.500817 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.500829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.500906 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.501031 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.501032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.501187 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.501255 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.501256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.501900 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.501900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.502393 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.502395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.502667 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.502668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.502718 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.503086 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.503091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.503424 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.504812 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.504813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.504813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.504971 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.505143 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.505143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.505144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.505200 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.505405 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.505406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.505451 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.505519 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.505623 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:26.528208 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.521196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.23.173.254:42487)
0812 14:27:26.521251 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.521253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.521278 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:27:26.521306 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.521307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.521316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.521356 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.521898 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.522406 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.523203 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.523422 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.523486 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.525801 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.525802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.525803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.525811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.527044 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.527374 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.527375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.527375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.527420 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.527670 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.527671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.527672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.527834 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.527939 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.528063 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.528133 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:26.529882 26171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:26.531661 26171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:26.537927 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.534148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:43273 (local address 127.0.0.1:37430)
0812 14:27:26.534275 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.534285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.534338 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.534857 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.534858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.534860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.534916 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.534918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.535894 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.535895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.536857 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.536860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.537695 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.537696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.537759 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.537771 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.537830 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.537867 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":53,"thread_start_us":87,"threads_started":1}
I20260812 14:27:26.537984 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.534391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.23.173.194:43273)
0812 14:27:26.534618 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.534623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.534634 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:26.534673 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.534675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.534690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.534997 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.535059 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.535801 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.536946 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.537613 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.537691 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.537747 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.537845 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.537883 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260812 14:27:26.538812 26629 tablet_service.cc:1467] Processing CreateTablet for tablet 827b365c75f4403aa31213eb3c657a75 (DEFAULT_TABLE table=test-table [id=1a30fbca81184008afbdb3c0cb3cb1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:26.539094 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.534328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:40651 (local address 127.0.0.1:57272)
0812 14:27:26.534428 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.534436 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.534480 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.534938 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.534939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.534941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.535000 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.535002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.536132 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.536133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.537184 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.537186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.538851 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.538852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.538915 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.538919 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.538980 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.539022 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":69,"threads_started":1}
I20260812 14:27:26.539179 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.533925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39007 (local address 127.0.0.1:49196)
0812 14:27:26.534054 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.534096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.534159 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.534444 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.534449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.534451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.534521 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.534523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.535809 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.535810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.537168 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.537171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.538985 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.538986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.539047 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.539050 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.539090 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.539124 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":49,"thread_start_us":65,"threads_started":1}
I20260812 14:27:26.539239 26629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827b365c75f4403aa31213eb3c657a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.539520 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.534038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.23.173.193:39007)
0812 14:27:26.534201 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.534206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.534217 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:26.534257 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.534259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.534275 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.534358 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.534580 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.535740 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.537258 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.537989 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.538986 (+ 997us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.539357 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.539385 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.539425 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260812 14:27:26.539697 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.534532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.23.173.195:40651)
0812 14:27:26.534730 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.534735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.534744 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:26.534781 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.534782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.534796 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.534870 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.535076 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.536055 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.538085 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.538770 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.538851 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.538909 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.539507 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.539547 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260812 14:27:26.540442 26882 tablet_service.cc:1467] Processing CreateTablet for tablet 827b365c75f4403aa31213eb3c657a75 (DEFAULT_TABLE table=test-table [id=1a30fbca81184008afbdb3c0cb3cb1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:26.540849 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827b365c75f4403aa31213eb3c657a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.543763 27027 tablet_bootstrap.cc:492] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Bootstrap starting.
I20260812 14:27:26.543856 26398 tablet_service.cc:1467] Processing CreateTablet for tablet 827b365c75f4403aa31213eb3c657a75 (DEFAULT_TABLE table=test-table [id=1a30fbca81184008afbdb3c0cb3cb1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:26.544217 27029 tablet_bootstrap.cc:492] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Bootstrap starting.
I20260812 14:27:26.544265 26398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827b365c75f4403aa31213eb3c657a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.544531 27027 tablet_bootstrap.cc:654] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.544891 27027 log.cc:824] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:26.544977 27029 tablet_bootstrap.cc:654] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.545280 27029 log.cc:824] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:26.545890 27027 tablet_bootstrap.cc:492] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: No bootstrap required, opened a new log
I20260812 14:27:26.545948 27029 tablet_bootstrap.cc:492] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: No bootstrap required, opened a new log
I20260812 14:27:26.545951 27027 ts_tablet_manager.cc:1397] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:26.546000 27029 ts_tablet_manager.cc:1397] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:26.547498 27029 raft_consensus.cc:348] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.547646 27029 raft_consensus.cc:374] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.547670 27029 raft_consensus.cc:729] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188fa031a286433da9e688b9992fb685, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.547828 27029 consensus_queue.cc:260] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.548000 27027 raft_consensus.cc:348] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.548123 27029 ts_tablet_manager.cc:1428] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:26.548156 27027 raft_consensus.cc:374] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.548189 27027 raft_consensus.cc:729] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cebd4169826461aaa88fd73cebb2f36, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.548275 26988 heartbeater.cc:503] Master 127.23.173.254:42487 was elected leader, sending a full tablet report...
I20260812 14:27:26.548290 27027 consensus_queue.cc:260] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.548542 27027 ts_tablet_manager.cc:1428] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:27:26.548664 26740 heartbeater.cc:503] Master 127.23.173.254:42487 was elected leader, sending a full tablet report...
I20260812 14:27:26.548807 27032 tablet_bootstrap.cc:492] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Bootstrap starting.
I20260812 14:27:26.549467 27032 tablet_bootstrap.cc:654] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.549767 27032 log.cc:824] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:26.550536 27032 tablet_bootstrap.cc:492] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: No bootstrap required, opened a new log
I20260812 14:27:26.550603 27032 ts_tablet_manager.cc:1397] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:26.552006 27032 raft_consensus.cc:348] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.552132 27032 raft_consensus.cc:374] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.552152 27032 raft_consensus.cc:729] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31517d827ca44133b78849003b79d7e9, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.552233 27032 consensus_queue.cc:260] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.552419 27032 ts_tablet_manager.cc:1428] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:26.552510 26511 heartbeater.cc:503] Master 127.23.173.254:42487 was elected leader, sending a full tablet report...
I20260812 14:27:26.562887 27034 raft_consensus.cc:482] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:26.563001 27034 raft_consensus.cc:504] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.563333 27034 leader_election.cc:288] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007), 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651)
I20260812 14:27:26.566074 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.563502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39007 (local address 127.23.173.194:59871)
0812 14:27:26.563553 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.563565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.563612 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.563801 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.563803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.563804 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.563881 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.563883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.564544 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.564545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.565294 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.565296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.565825 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.565826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.565901 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.565904 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.565950 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.566000 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:27:26.566074 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.563578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:59871 (local address 127.23.173.193:39007)
0812 14:27:26.563635 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.563637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.563647 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:26.563675 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.563676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.563685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.563779 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.563957 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.564450 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.565384 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.565759 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.565822 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.565831 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.565966 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.566007 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:27:26.566311 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.563679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:40651 (local address 127.23.173.194:59271)
0812 14:27:26.563887 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.563896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.563954 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.564077 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.564077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.564079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.564160 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.564162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.564781 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.564782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.565516 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.565518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.566049 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.566050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.566125 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.566129 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.566176 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.566219 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":153,"threads_started":1}
I20260812 14:27:26.566448 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.563801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:59271 (local address 127.23.173.195:40651)
0812 14:27:26.563862 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.563864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.563961 (+ 97us) server_negotiation.cc:407] Connection header received
0812 14:27:26.564008 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.564009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.564017 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.564064 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.564221 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.564704 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.565604 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.565967 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.566036 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.566044 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.566286 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.566329 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:27:26.566403 26431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827b365c75f4403aa31213eb3c657a75" candidate_uuid: "3cebd4169826461aaa88fd73cebb2f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31517d827ca44133b78849003b79d7e9" is_pre_election: true
I20260812 14:27:26.566552 26431 raft_consensus.cc:2454] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebd4169826461aaa88fd73cebb2f36 in term 0.
I20260812 14:27:26.566794 26557 leader_election.cc:302] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31517d827ca44133b78849003b79d7e9, 3cebd4169826461aaa88fd73cebb2f36; no voters:
I20260812 14:27:26.566884 26918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827b365c75f4403aa31213eb3c657a75" candidate_uuid: "3cebd4169826461aaa88fd73cebb2f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188fa031a286433da9e688b9992fb685" is_pre_election: true
I20260812 14:27:26.566933 27034 raft_consensus.cc:2781] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:26.566978 27034 raft_consensus.cc:482] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:26.566995 27034 raft_consensus.cc:3037] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.567034 26918 raft_consensus.cc:2454] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebd4169826461aaa88fd73cebb2f36 in term 0.
I20260812 14:27:26.567803 27034 raft_consensus.cc:504] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.567914 27034 leader_election.cc:288] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [CANDIDATE]: Term 1 election: Requested vote from peers 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007), 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651)
I20260812 14:27:26.568081 26431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827b365c75f4403aa31213eb3c657a75" candidate_uuid: "3cebd4169826461aaa88fd73cebb2f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31517d827ca44133b78849003b79d7e9"
I20260812 14:27:26.568084 26918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827b365c75f4403aa31213eb3c657a75" candidate_uuid: "3cebd4169826461aaa88fd73cebb2f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188fa031a286433da9e688b9992fb685"
I20260812 14:27:26.568153 26431 raft_consensus.cc:3037] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.568157 26918 raft_consensus.cc:3037] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.568826 26431 raft_consensus.cc:2454] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cebd4169826461aaa88fd73cebb2f36 in term 1.
I20260812 14:27:26.568915 26918 raft_consensus.cc:2454] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cebd4169826461aaa88fd73cebb2f36 in term 1.
I20260812 14:27:26.569003 26557 leader_election.cc:302] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31517d827ca44133b78849003b79d7e9, 3cebd4169826461aaa88fd73cebb2f36; no voters:
I20260812 14:27:26.569095 27034 raft_consensus.cc:2781] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:26.569255 27034 raft_consensus.cc:686] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 LEADER]: Becoming Leader. State: Replica: 3cebd4169826461aaa88fd73cebb2f36, State: Running, Role: LEADER
I20260812 14:27:26.569355 27034 consensus_queue.cc:237] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.570226 26172 catalog_manager.cc:5495] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194). New cstate: current_term: 1 leader_uuid: "3cebd4169826461aaa88fd73cebb2f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:26.586251 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.583181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.23.173.194:43273)
0812 14:27:26.583240 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.583242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.583271 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:27:26.583366 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.583367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.583385 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.583444 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:26.583571 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.584106 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.584961 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.585185 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.585258 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.585521 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.585634 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.585711 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.585819 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.585844 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 14:27:26.586526 26742 tablet.cc:2369] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:27:26.588366 26431 raft_consensus.cc:1261] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Refusing update from remote peer 3cebd4169826461aaa88fd73cebb2f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:26.588366 26918 raft_consensus.cc:1261] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Refusing update from remote peer 3cebd4169826461aaa88fd73cebb2f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:26.588745 27039 consensus_queue.cc:1035] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:26.588896 27034 consensus_queue.cc:1035] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:26.591921 27044 mvcc.cc:205] Tried to move back new op lower bound from 7317687691622576128 to 7317687691548708864. Current Snapshot: MvccSnapshot[applied={T|T < 7317687691622576128}]
I20260812 14:27:26.592374 27046 mvcc.cc:205] Tried to move back new op lower bound from 7317687691622576128 to 7317687691548708864. Current Snapshot: MvccSnapshot[applied={T|T < 7317687691622576128}]
I20260812 14:27:26.597147 26172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:26.597286 26172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:26.598223 26172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: Sending DeleteTablet for 3 replicas of tablet 827b365c75f4403aa31213eb3c657a75
I20260812 14:27:26.598618 26398 tablet_service.cc:1514] Processing DeleteTablet for tablet 827b365c75f4403aa31213eb3c657a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:26 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:49196
I20260812 14:27:26.598658 26629 tablet_service.cc:1514] Processing DeleteTablet for tablet 827b365c75f4403aa31213eb3c657a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:26 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:37430
I20260812 14:27:26.598745 26882 tablet_service.cc:1514] Processing DeleteTablet for tablet 827b365c75f4403aa31213eb3c657a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:26 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:57272
I20260812 14:27:26.598821 26172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:26.598973 26172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:26.599015 27064 tablet_replica.cc:330] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: stopping tablet replica
I20260812 14:27:26.599231 27064 raft_consensus.cc:2229] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:26.599375 27064 raft_consensus.cc:2258] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:26.599545 27063 tablet_replica.cc:330] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: stopping tablet replica
I20260812 14:27:26.599646 27063 raft_consensus.cc:2229] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:26.600116 27062 tablet_replica.cc:330] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: stopping tablet replica
I20260812 14:27:26.600239 27062 raft_consensus.cc:2229] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:26.600360 27062 raft_consensus.cc:2258] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:26.599797 27063 raft_consensus.cc:2258] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:26.600497 26629 tablet_service.cc:1467] Processing CreateTablet for tablet a93ee9cd4b044d6683733261bc5f66c3 (DEFAULT_TABLE table=test-table [id=4a206377402648e1b8a5f61ad9b2fa7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:26.600510 26882 tablet_service.cc:1467] Processing CreateTablet for tablet a93ee9cd4b044d6683733261bc5f66c3 (DEFAULT_TABLE table=test-table [id=4a206377402648e1b8a5f61ad9b2fa7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:26.600644 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ee9cd4b044d6683733261bc5f66c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.600644 26629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ee9cd4b044d6683733261bc5f66c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.601055 27062 ts_tablet_manager.cc:1905] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:26.602037 27027 tablet_bootstrap.cc:492] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Bootstrap starting.
I20260812 14:27:26.602113 27029 tablet_bootstrap.cc:492] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Bootstrap starting.
I20260812 14:27:26.602460 27027 tablet_bootstrap.cc:654] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.602506 26398 tablet_service.cc:1467] Processing CreateTablet for tablet a93ee9cd4b044d6683733261bc5f66c3 (DEFAULT_TABLE table=test-table [id=4a206377402648e1b8a5f61ad9b2fa7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:26.602612 26398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ee9cd4b044d6683733261bc5f66c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.602663 27029 tablet_bootstrap.cc:654] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.603305 27063 ts_tablet_manager.cc:1905] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:26.603978 27032 tablet_bootstrap.cc:492] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Bootstrap starting.
I20260812 14:27:26.604179 27064 ts_tablet_manager.cc:1905] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:26.604403 27032 tablet_bootstrap.cc:654] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.605820 27063 ts_tablet_manager.cc:1918] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:26.605903 27063 log.cc:1196] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/827b365c75f4403aa31213eb3c657a75
I20260812 14:27:26.606055 27064 ts_tablet_manager.cc:1918] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:26.606163 27064 log.cc:1196] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/827b365c75f4403aa31213eb3c657a75
I20260812 14:27:26.606186 27063 ts_tablet_manager.cc:1939] T 827b365c75f4403aa31213eb3c657a75 P 3cebd4169826461aaa88fd73cebb2f36: Deleting consensus metadata
I20260812 14:27:26.606455 27064 ts_tablet_manager.cc:1939] T 827b365c75f4403aa31213eb3c657a75 P 188fa031a286433da9e688b9992fb685: Deleting consensus metadata
I20260812 14:27:26.606655 26151 catalog_manager.cc:4841] TS 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273): tablet 827b365c75f4403aa31213eb3c657a75 (table test-table [id=1a30fbca81184008afbdb3c0cb3cb1e2]) successfully deleted
I20260812 14:27:26.606918 26151 catalog_manager.cc:4841] TS 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651): tablet 827b365c75f4403aa31213eb3c657a75 (table test-table [id=1a30fbca81184008afbdb3c0cb3cb1e2]) successfully deleted
I20260812 14:27:26.606912 27062 ts_tablet_manager.cc:1918] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:26.607013 27062 log.cc:1196] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/827b365c75f4403aa31213eb3c657a75
I20260812 14:27:26.607247 27062 ts_tablet_manager.cc:1939] T 827b365c75f4403aa31213eb3c657a75 P 31517d827ca44133b78849003b79d7e9: Deleting consensus metadata
I20260812 14:27:26.607653 26151 catalog_manager.cc:4841] TS 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007): tablet 827b365c75f4403aa31213eb3c657a75 (table test-table [id=1a30fbca81184008afbdb3c0cb3cb1e2]) successfully deleted
I20260812 14:27:26.608036 27027 tablet_bootstrap.cc:492] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: No bootstrap required, opened a new log
I20260812 14:27:26.608142 27027 ts_tablet_manager.cc:1397] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260812 14:27:26.608192 27029 tablet_bootstrap.cc:492] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: No bootstrap required, opened a new log
I20260812 14:27:26.608237 27029 ts_tablet_manager.cc:1397] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260812 14:27:26.608330 27032 tablet_bootstrap.cc:492] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: No bootstrap required, opened a new log
I20260812 14:27:26.608369 27032 ts_tablet_manager.cc:1397] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260812 14:27:26.608400 27029 raft_consensus.cc:348] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.608490 27029 raft_consensus.cc:374] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.608510 27029 raft_consensus.cc:729] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188fa031a286433da9e688b9992fb685, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.608399 27027 raft_consensus.cc:348] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.608541 27027 raft_consensus.cc:374] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.608511 27032 raft_consensus.cc:348] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.608587 27032 raft_consensus.cc:374] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.608562 27029 consensus_queue.cc:260] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.608563 27027 raft_consensus.cc:729] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cebd4169826461aaa88fd73cebb2f36, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.608608 27032 raft_consensus.cc:729] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31517d827ca44133b78849003b79d7e9, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.608657 27029 ts_tablet_manager.cc:1428] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:26.608650 27032 consensus_queue.cc:260] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.608636 27027 consensus_queue.cc:260] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.608726 27032 ts_tablet_manager.cc:1428] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:26.608762 27027 ts_tablet_manager.cc:1428] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260812 14:27:26.671201 26513 tablet.cc:2369] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:27:26.711282 27033 raft_consensus.cc:482] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:26.711391 27033 raft_consensus.cc:504] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.711903 27033 leader_election.cc:288] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007), 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273)
I20260812 14:27:26.715601 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.712232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:43273 (local address 127.23.173.195:36307)
0812 14:27:26.712382 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.712391 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.712431 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.712607 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.712608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.712609 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.712698 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.712699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.713629 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.713631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.714736 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.714738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.715327 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.715329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.715403 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.715406 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.715454 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.715496 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260812 14:27:26.715595 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.712452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:36307 (local address 127.23.173.194:43273)
0812 14:27:26.712512 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.712514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.712520 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:27:26.712544 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.712545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.712553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.712603 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.712808 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.713525 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.714842 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.715220 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.715286 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.715296 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.715472 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.715511 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:27:26.715819 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.712110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39007 (local address 127.23.173.195:36543)
0812 14:27:26.712161 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.712174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.712226 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.712396 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.712398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.712400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.712480 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.712482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.713376 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.713378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.714694 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.714697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.715533 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.715534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.715634 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.715638 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.715682 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.715746 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:27:26.715917 26670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ee9cd4b044d6683733261bc5f66c3" candidate_uuid: "188fa031a286433da9e688b9992fb685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebd4169826461aaa88fd73cebb2f36" is_pre_election: true
I20260812 14:27:26.716060 26670 raft_consensus.cc:2454] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188fa031a286433da9e688b9992fb685 in term 0.
I20260812 14:27:26.716107 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.712193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:36543 (local address 127.23.173.193:39007)
0812 14:27:26.712249 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.712251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.712263 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:27:26.712293 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.712294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.712303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.712376 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.712582 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.713263 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.714817 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.715410 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.715489 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.715496 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.715993 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.716042 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:27:26.716248 26797 leader_election.cc:302] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188fa031a286433da9e688b9992fb685, 3cebd4169826461aaa88fd73cebb2f36; no voters:
I20260812 14:27:26.716274 26431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ee9cd4b044d6683733261bc5f66c3" candidate_uuid: "188fa031a286433da9e688b9992fb685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31517d827ca44133b78849003b79d7e9" is_pre_election: true
I20260812 14:27:26.716341 26431 raft_consensus.cc:2454] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188fa031a286433da9e688b9992fb685 in term 0.
I20260812 14:27:26.716384 27033 raft_consensus.cc:2781] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:26.716423 27033 raft_consensus.cc:482] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:26.716447 27033 raft_consensus.cc:3037] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.717280 27033 raft_consensus.cc:504] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.717412 27033 leader_election.cc:288] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [CANDIDATE]: Term 1 election: Requested vote from peers 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007), 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273)
I20260812 14:27:26.717737 26431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ee9cd4b044d6683733261bc5f66c3" candidate_uuid: "188fa031a286433da9e688b9992fb685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31517d827ca44133b78849003b79d7e9"
I20260812 14:27:26.717832 26431 raft_consensus.cc:3037] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.718586 26431 raft_consensus.cc:2454] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188fa031a286433da9e688b9992fb685 in term 1.
I20260812 14:27:26.718708 26670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ee9cd4b044d6683733261bc5f66c3" candidate_uuid: "188fa031a286433da9e688b9992fb685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebd4169826461aaa88fd73cebb2f36"
I20260812 14:27:26.718786 26670 raft_consensus.cc:3037] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.718811 26797 leader_election.cc:302] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188fa031a286433da9e688b9992fb685, 31517d827ca44133b78849003b79d7e9; no voters:
I20260812 14:27:26.718914 27033 raft_consensus.cc:2781] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:26.719074 27033 raft_consensus.cc:686] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 1 LEADER]: Becoming Leader. State: Replica: 188fa031a286433da9e688b9992fb685, State: Running, Role: LEADER
I20260812 14:27:26.719151 27033 consensus_queue.cc:237] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.719753 26670 raft_consensus.cc:2454] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188fa031a286433da9e688b9992fb685 in term 1.
I20260812 14:27:26.720134 26170 catalog_manager.cc:5495] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 reported cstate change: term changed from 0 to 1, leader changed from <none> to 188fa031a286433da9e688b9992fb685 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "188fa031a286433da9e688b9992fb685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } health_report { overall_health: HEALTHY } } }
W20260812 14:27:26.747566 26990 tablet.cc:2369] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544845, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:26.763530 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.756576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42487 (local address 127.0.0.1:50798)
0812 14:27:26.756748 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.756759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.756835 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.756972 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.756973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.757323 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:26.757377 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.757379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.758035 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.758036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.758545 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.758547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.758860 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.758861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.758911 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.759429 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:26.759433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:26.760919 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:26.762493 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.762501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.762504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.762726 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.762927 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:26.762928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:26.762928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:26.762984 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:26.763210 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:26.763217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:26.763265 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.763315 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.763450 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260812 14:27:26.763607 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.756652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.23.173.254:42487)
0812 14:27:26.756740 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.756743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.756829 (+ 86us) server_negotiation.cc:407] Connection header received
0812 14:27:26.756898 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.756899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.756909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.756957 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.757467 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.757959 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.758624 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.758799 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.758852 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.761014 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.761016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.761016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.761023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.762415 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.762809 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.762809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.762810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.762873 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.763037 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.763037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.763038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.763153 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.763203 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.763337 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.763507 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":61}
I20260812 14:27:26.765798 26170 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:50798:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:27:26.766089 26170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:26.767520 26398 tablet_service.cc:1467] Processing CreateTablet for tablet ef5f24d6235b455e93d88bd50c908451 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa40bd45e44484aaf0d472502d4af9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:27:26.767558 26629 tablet_service.cc:1467] Processing CreateTablet for tablet ef5f24d6235b455e93d88bd50c908451 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa40bd45e44484aaf0d472502d4af9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:27:26.767607 26882 tablet_service.cc:1467] Processing CreateTablet for tablet ef5f24d6235b455e93d88bd50c908451 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa40bd45e44484aaf0d472502d4af9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:27:26.767691 26398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5f24d6235b455e93d88bd50c908451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.767740 26629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5f24d6235b455e93d88bd50c908451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.767799 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5f24d6235b455e93d88bd50c908451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:26.769150 27032 tablet_bootstrap.cc:492] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: Bootstrap starting.
I20260812 14:27:26.769297 27027 tablet_bootstrap.cc:492] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: Bootstrap starting.
I20260812 14:27:26.769297 27029 tablet_bootstrap.cc:492] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: Bootstrap starting.
I20260812 14:27:26.769556 27032 tablet_bootstrap.cc:654] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.769644 27027 tablet_bootstrap.cc:654] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.769704 27029 tablet_bootstrap.cc:654] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:26.770190 27032 tablet_bootstrap.cc:492] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: No bootstrap required, opened a new log
I20260812 14:27:26.770251 27032 ts_tablet_manager.cc:1397] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:26.770309 27027 tablet_bootstrap.cc:492] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: No bootstrap required, opened a new log
I20260812 14:27:26.770360 27027 ts_tablet_manager.cc:1397] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:26.770431 27032 raft_consensus.cc:348] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.770465 27027 raft_consensus.cc:348] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.770515 27032 raft_consensus.cc:374] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.770521 27027 raft_consensus.cc:374] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.770546 27027 raft_consensus.cc:729] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cebd4169826461aaa88fd73cebb2f36, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.770546 27032 raft_consensus.cc:729] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31517d827ca44133b78849003b79d7e9, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.770587 27027 consensus_queue.cc:260] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.770587 27032 consensus_queue.cc:260] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.770601 27029 tablet_bootstrap.cc:492] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: No bootstrap required, opened a new log
I20260812 14:27:26.770651 27029 ts_tablet_manager.cc:1397] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:26.770677 27032 ts_tablet_manager.cc:1428] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:26.770677 27027 ts_tablet_manager.cc:1428] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.770790 27034 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.770798 27029 raft_consensus.cc:348] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.770862 27029 raft_consensus.cc:374] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:26.770881 27029 raft_consensus.cc:729] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188fa031a286433da9e688b9992fb685, State: Initialized, Role: FOLLOWER
I20260812 14:27:26.770866 27034 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.770848 27037 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.770931 27037 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.770925 27029 consensus_queue.cc:260] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.771005 27029 ts_tablet_manager.cc:1428] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:26.771185 27033 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.771281 27033 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.865314 27037 raft_consensus.cc:482] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:26.865440 27037 raft_consensus.cc:504] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.865769 27037 leader_election.cc:288] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273), 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651)
I20260812 14:27:26.868896 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.866297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:58547 (local address 127.23.173.195:40651)
0812 14:27:26.866381 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.866383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.866391 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:26.866419 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.866420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.866430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.866478 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.866669 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.867153 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.868211 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.868571 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.868634 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.868644 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.868782 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.868822 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 14:27:26.868907 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.866096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:53209 (local address 127.23.173.194:43273)
0812 14:27:26.866171 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.866174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.866227 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:27:26.866341 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.866342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.866352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.866381 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:26.866586 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.867112 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.868203 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.868577 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.868643 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.868653 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.868794 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.868840 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:27:26.868933 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.866201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:40651 (local address 127.23.173.193:58547)
0812 14:27:26.866298 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.866307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.866362 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.866490 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.866491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.866493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.866593 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.866595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.867236 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.867237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.868118 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.868121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.868650 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.868651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.868713 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.868717 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.868758 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.868807 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":34,"thread_start_us":60,"threads_started":1}
I20260812 14:27:26.868935 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.865991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:43273 (local address 127.23.173.193:53209)
0812 14:27:26.866149 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.866162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.866248 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.866443 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.866444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.866445 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:26.866521 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.866524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.867202 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.867203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.868119 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.868122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.868653 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.868654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.868727 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.868730 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.868782 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.868828 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":47,"thread_start_us":82,"threads_started":1}
I20260812 14:27:26.869266 26918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5f24d6235b455e93d88bd50c908451" candidate_uuid: "31517d827ca44133b78849003b79d7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188fa031a286433da9e688b9992fb685" is_pre_election: true
I20260812 14:27:26.869382 26918 raft_consensus.cc:2454] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31517d827ca44133b78849003b79d7e9 in term 0.
I20260812 14:27:26.869418 26670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5f24d6235b455e93d88bd50c908451" candidate_uuid: "31517d827ca44133b78849003b79d7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebd4169826461aaa88fd73cebb2f36" is_pre_election: true
I20260812 14:27:26.869498 26670 raft_consensus.cc:2454] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31517d827ca44133b78849003b79d7e9 in term 0.
I20260812 14:27:26.869601 26320 leader_election.cc:302] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188fa031a286433da9e688b9992fb685, 31517d827ca44133b78849003b79d7e9; no voters:
I20260812 14:27:26.869733 27037 raft_consensus.cc:2781] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:26.869779 27037 raft_consensus.cc:482] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:26.869792 27037 raft_consensus.cc:3037] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.870494 27037 raft_consensus.cc:504] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.870605 27037 leader_election.cc:288] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273), 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651)
I20260812 14:27:26.870790 26670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5f24d6235b455e93d88bd50c908451" candidate_uuid: "31517d827ca44133b78849003b79d7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebd4169826461aaa88fd73cebb2f36"
I20260812 14:27:26.870788 26918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5f24d6235b455e93d88bd50c908451" candidate_uuid: "31517d827ca44133b78849003b79d7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188fa031a286433da9e688b9992fb685"
I20260812 14:27:26.870865 26918 raft_consensus.cc:3037] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.870867 26670 raft_consensus.cc:3037] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:26.871565 26670 raft_consensus.cc:2454] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31517d827ca44133b78849003b79d7e9 in term 1.
I20260812 14:27:26.871753 26320 leader_election.cc:302] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31517d827ca44133b78849003b79d7e9, 3cebd4169826461aaa88fd73cebb2f36; no voters:
I20260812 14:27:26.871853 27037 raft_consensus.cc:2781] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:26.871960 27037 raft_consensus.cc:686] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 31517d827ca44133b78849003b79d7e9, State: Running, Role: LEADER
I20260812 14:27:26.872038 27037 consensus_queue.cc:237] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } }
I20260812 14:27:26.872073 26918 raft_consensus.cc:2454] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31517d827ca44133b78849003b79d7e9 in term 1.
I20260812 14:27:26.872400 27090 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: TxnStatusTablet state changed. Reason: New leader 31517d827ca44133b78849003b79d7e9. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.872494 27090 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:27:26.872727 27093 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:27:26.872840 27093 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:27:26.872880 26171 catalog_manager.cc:5495] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31517d827ca44133b78849003b79d7e9 (127.23.173.193). New cstate: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:26.891036 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.888002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39007 (local address 127.0.0.1:49210)
0812 14:27:26.888128 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.888142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.888201 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.888395 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.888395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.888396 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:26.888448 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.888450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.889398 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.889399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.889995 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.889996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.890470 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.890471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.890530 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.890548 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.890860 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.890872 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.890921 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.890972 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 14:27:26.891040 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.888112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.23.173.193:39007)
0812 14:27:26.888181 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.888183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.888195 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:27:26.888273 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.888274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.888298 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.888385 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:26.888507 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.889275 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.890072 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.890391 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.890457 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.890634 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.890770 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.890847 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.890927 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.890953 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:27:26.892421 26918 raft_consensus.cc:1261] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Refusing update from remote peer 31517d827ca44133b78849003b79d7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:26.892508 26670 raft_consensus.cc:1261] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Refusing update from remote peer 31517d827ca44133b78849003b79d7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:26.892767 27090 consensus_queue.cc:1035] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:26.892916 27037 consensus_queue.cc:1035] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:26.893519 27033 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: TxnStatusTablet state changed. Reason: New leader 31517d827ca44133b78849003b79d7e9. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.893625 27033 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.893571 27034 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: TxnStatusTablet state changed. Reason: New leader 31517d827ca44133b78849003b79d7e9. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.893671 27034 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.894663 27090 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.894764 27090 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:27:26.894683 27100 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.894814 27100 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 31517d827ca44133b78849003b79d7e9: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:27:26.895351 27047 mvcc.cc:205] Tried to move back new op lower bound from 7317687692868886528 to 7317687692788305920. Current Snapshot: MvccSnapshot[applied={T|T < 7317687692868886528}]
I20260812 14:27:26.895447 27033 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.895540 27033 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 188fa031a286433da9e688b9992fb685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.895776 27034 tablet_replica.cc:439] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31517d827ca44133b78849003b79d7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 } } peers { permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 } } peers { permanent_uuid: "188fa031a286433da9e688b9992fb685" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 40651 } } }
I20260812 14:27:26.895866 27034 tablet_replica.cc:442] T ef5f24d6235b455e93d88bd50c908451 P 3cebd4169826461aaa88fd73cebb2f36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:26.899946 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.897188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.23.173.195:40651)
0812 14:27:26.897225 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.897227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.897236 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:26.897293 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.897294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.897310 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.897375 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:26.897499 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.898046 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.899117 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.899361 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.899415 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.899606 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.899759 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.899830 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.899865 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.899887 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:27:26.905390 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.902661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39007 (local address 127.23.173.195:44497)
0812 14:27:26.902748 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.902760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.902798 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.902928 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.902929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.902930 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:26.902986 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.902988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.903910 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.903911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.904493 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.904496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.904957 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.904958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.905009 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.905033 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.905228 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.905231 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.905277 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.905324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 14:27:26.905401 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.902727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:44497 (local address 127.23.173.193:39007)
0812 14:27:26.902779 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.902781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.902789 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:26.902846 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.902846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.902857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.902925 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:26.903031 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.903816 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.904572 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.904883 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.904943 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.905108 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.905144 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.905192 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.905290 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.905314 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:27:26.908198 26670 raft_consensus.cc:1261] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Refusing update from remote peer 188fa031a286433da9e688b9992fb685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:26.908246 26431 raft_consensus.cc:1261] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Refusing update from remote peer 188fa031a286433da9e688b9992fb685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:26.908522 27033 consensus_queue.cc:1035] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebd4169826461aaa88fd73cebb2f36" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:26.908686 27073 consensus_queue.cc:1035] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31517d827ca44133b78849003b79d7e9" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:26.918411 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.915450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:46177 (local address 127.23.173.195:40651)
0812 14:27:26.915490 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.915493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.915502 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:26.915598 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.915598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.915608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.915665 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:26.915833 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.916589 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.917760 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.917977 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.918035 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.918213 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.918243 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.918272 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.918328 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.918347 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:27:26.918483 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.915210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:40651 (local address 127.23.173.193:46177)
0812 14:27:26.915406 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:26.915419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:26.915525 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:26.915654 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:26.915655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:26.915656 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:26.915752 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.915754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.916685 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.916686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.917663 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:26.917666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.918046 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.918047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:26.918116 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.918135 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:26.918276 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:26.918278 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:27:26.918326 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:27:26.918374 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260812 14:27:26.927208 26170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:26.927340 26170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:26.928040 26170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61ba4a5dc4f4ccbac821de379750f1b: Sending DeleteTablet for 3 replicas of tablet a93ee9cd4b044d6683733261bc5f66c3
I20260812 14:27:26.928321 26398 tablet_service.cc:1514] Processing DeleteTablet for tablet a93ee9cd4b044d6683733261bc5f66c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:26 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:49196
I20260812 14:27:26.928450 27062 tablet_replica.cc:330] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: stopping tablet replica
I20260812 14:27:26.928503 27062 raft_consensus.cc:2229] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:26.928546 27062 raft_consensus.cc:2258] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:26.928527 26629 tablet_service.cc:1514] Processing DeleteTablet for tablet a93ee9cd4b044d6683733261bc5f66c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:26 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:37430
I20260812 14:27:26.928620 26882 tablet_service.cc:1514] Processing DeleteTablet for tablet a93ee9cd4b044d6683733261bc5f66c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:26 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:57272
I20260812 14:27:26.928682 27063 tablet_replica.cc:330] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: stopping tablet replica
I20260812 14:27:26.928707 27064 tablet_replica.cc:330] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: stopping tablet replica
I20260812 14:27:26.928745 27064 raft_consensus.cc:2229] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:26.928750 27063 raft_consensus.cc:2229] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:26.928785 27063 raft_consensus.cc:2258] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:26.928864 27064 raft_consensus.cc:2258] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:26.929157 27062 ts_tablet_manager.cc:1905] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:26.929886 27063 ts_tablet_manager.cc:1905] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:26.931143 27062 ts_tablet_manager.cc:1918] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:26.931205 27062 log.cc:1196] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/a93ee9cd4b044d6683733261bc5f66c3
I20260812 14:27:26.931447 27062 ts_tablet_manager.cc:1939] T a93ee9cd4b044d6683733261bc5f66c3 P 31517d827ca44133b78849003b79d7e9: Deleting consensus metadata
I20260812 14:27:26.935652 26151 catalog_manager.cc:4841] TS 31517d827ca44133b78849003b79d7e9 (127.23.173.193:39007): tablet a93ee9cd4b044d6683733261bc5f66c3 (table test-table [id=4a206377402648e1b8a5f61ad9b2fa7d]) successfully deleted
I20260812 14:27:26.931675 27063 ts_tablet_manager.cc:1918] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:26.935990 27063 log.cc:1196] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/a93ee9cd4b044d6683733261bc5f66c3
I20260812 14:27:26.936108 27064 ts_tablet_manager.cc:1905] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:26.936242 27063 ts_tablet_manager.cc:1939] T a93ee9cd4b044d6683733261bc5f66c3 P 3cebd4169826461aaa88fd73cebb2f36: Deleting consensus metadata
I20260812 14:27:26.937887 27064 ts_tablet_manager.cc:1918] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:26.937963 27064 log.cc:1196] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/a93ee9cd4b044d6683733261bc5f66c3
I20260812 14:27:26.938182 27064 ts_tablet_manager.cc:1939] T a93ee9cd4b044d6683733261bc5f66c3 P 188fa031a286433da9e688b9992fb685: Deleting consensus metadata
I20260812 14:27:26.938455 26151 catalog_manager.cc:4841] TS 3cebd4169826461aaa88fd73cebb2f36 (127.23.173.194:43273): tablet a93ee9cd4b044d6683733261bc5f66c3 (table test-table [id=4a206377402648e1b8a5f61ad9b2fa7d]) successfully deleted
I20260812 14:27:26.938820 26151 catalog_manager.cc:4841] TS 188fa031a286433da9e688b9992fb685 (127.23.173.195:40651): tablet a93ee9cd4b044d6683733261bc5f66c3 (table test-table [id=4a206377402648e1b8a5f61ad9b2fa7d]) successfully deleted
Aug 12 14:27:26 dist-test-slave-1l3l krb5kdc[26098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:26.947576 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.930874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.23.173.193:39007)
0812 14:27:26.930949 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.930952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.936783 (+ 5831us) server_negotiation.cc:407] Connection header received
0812 14:27:26.936907 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.936908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.936920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.936950 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.937719 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.938474 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.939818 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.940146 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.940226 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.944135 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.944143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.944143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.944152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.946210 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.946662 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.946665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.946666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.946733 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.946977 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.946978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.946978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.947160 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.947290 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.947397 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.947485 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 14:27:26.947966 26498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49224
I20260812 14:27:26.957110 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:26.949536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.23.173.254:42487)
0812 14:27:26.949589 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:26.949590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:26.949721 (+ 131us) server_negotiation.cc:407] Connection header received
0812 14:27:26.949760 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:26.949761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:26.949774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:26.949836 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:26.950440 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.951211 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.952273 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:26.952577 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:26.952652 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:26.954099 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:26.954101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:26.954101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:26.954109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:26.955800 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.956220 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.956221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.956221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.956291 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:26.956494 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:26.956495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:26.956496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:26.956649 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:26.956764 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:26.956921 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 14:27:26.957014 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260812 14:27:26.957486 26170 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50810
I20260812 14:27:26.958468 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26287
I20260812 14:27:26.970768 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26524
I20260812 14:27:26.980890 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26760
I20260812 14:27:26.998488 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26120
2026-08-12T14:27:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2496 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:27 dist-test-slave-1l3l krb5kdc[27129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:27 dist-test-slave-1l3l krb5kdc[27129](info): set up 2 sockets
Aug 12 14:27:27 dist-test-slave-1l3l krb5kdc[27129](info): commencing operation
krb5kdc: starting...
W20260812 14:27:28.200495 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:28.320444 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41657
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:28.401130 27150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:28.401335 27150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:28.401355 27150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:28.402464 27150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:28.402498 27150 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:27:28.402509 27150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:28.402521 27150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:28.402541 27150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:28.402556 27150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:28.403977 27150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:41657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:28.404167 27150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:28.404343 27150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:28.409109 27157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.409132 27158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.409158 27162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:28.409277 27150 server_base.cc:1034] running on GCE node
I20260812 14:27:28.409507 27150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:28.409791 27150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:28.410933 27150 hybrid_clock.cc:648] HybridClock initialized: now 1786544848410921 us; error 28 us; skew 500 ppm
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:28.413414 27150 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:28.413829 27150 webserver.cc:460] Webserver started at http://127.23.173.254:42537/ using document root <none> and password file <none>
I20260812 14:27:28.414004 27150 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:28.414045 27150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:28.414132 27150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:28.415112 27150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "bc89ea4c875d40e9a8ba84f7083cd51c"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "c641ca7ba13e521f345cb443e61d39e5"
server_key_iv: "91c2bf6a64a131286a7b2b4b369cd362"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.415423 27150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "bc89ea4c875d40e9a8ba84f7083cd51c"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "c641ca7ba13e521f345cb443e61d39e5"
server_key_iv: "91c2bf6a64a131286a7b2b4b369cd362"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.416704 27150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:28.417332 27175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.417490 27150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:28.417563 27150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "bc89ea4c875d40e9a8ba84f7083cd51c"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "c641ca7ba13e521f345cb443e61d39e5"
server_key_iv: "91c2bf6a64a131286a7b2b4b369cd362"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.417614 27150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:28.426422 27150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:28.428483 27150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:28.428589 27150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:28.433002 27150 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:41657
I20260812 14:27:28.433071 27272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:41657 every 8 connection(s)
I20260812 14:27:28.433357 27150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:28.433954 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:28.435194 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27150
I20260812 14:27:28.435267 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:28.435441 24247 external_mini_cluster.cc:1428] Setting key ec6be0518b1478351e769e69cc3713cf
I20260812 14:27:28.436547 27274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Bootstrap starting.
I20260812 14:27:28.437211 27274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:28.437477 27274 log.cc:824] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:28.438230 27274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: No bootstrap required, opened a new log
I20260812 14:27:28.440515 27274 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41657 } }
I20260812 14:27:28.440650 27274 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:28.440683 27274 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc89ea4c875d40e9a8ba84f7083cd51c, State: Initialized, Role: FOLLOWER
I20260812 14:27:28.440809 27274 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41657 } }
I20260812 14:27:28.440884 27274 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:28.440914 27274 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:28.440961 27274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:28.441509 27274 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41657 } }
I20260812 14:27:28.441603 27274 leader_election.cc:302] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc89ea4c875d40e9a8ba84f7083cd51c; no voters:
I20260812 14:27:28.441758 27274 leader_election.cc:288] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:28.441824 27284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:28.441995 27284 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [term 1 LEADER]: Becoming Leader. State: Replica: bc89ea4c875d40e9a8ba84f7083cd51c, State: Running, Role: LEADER
I20260812 14:27:28.442027 27274 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:28.442111 27284 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41657 } }
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:28.442461 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc89ea4c875d40e9a8ba84f7083cd51c. Latest consensus state: current_term: 1 leader_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41657 } } }
I20260812 14:27:28.442549 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:28.442469 27287 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc89ea4c875d40e9a8ba84f7083cd51c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41657 } } }
I20260812 14:27:28.442677 27287 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c [sys.catalog]: This master's current role is: LEADER
W20260812 14:27:28.443903 27315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:28.443964 27315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:28.444018 27297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:28.444185 27297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:28.445739 27297 catalog_manager.cc:1347] Generated new cluster ID: d50d7921c93f41fc9e78af5bb036458f
I20260812 14:27:28.445796 27297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:28.446563 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.435968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.23.173.254:41657)
0812 14:27:28.436169 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.436179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.436213 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:27:28.436720 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.436729 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.436789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.436947 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.437546 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.438829 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.439550 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.440219 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.440334 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.443113 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.443118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.443118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.443131 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.444928 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.445357 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.445361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.445361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.445467 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.445677 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.445678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.445679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.445967 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.446099 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.446223 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.446356 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260812 14:27:28.456884 27297 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:28.457382 27297 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:28.461133 27297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Generated new TSK 0
I20260812 14:27:28.461298 27297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:28.496800 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41657
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:28.604318 27323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:28.604581 27323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:28.604663 27323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:28.606300 27323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:28.606355 27323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:28.606382 27323 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:27:28.606405 27323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:27:28.606429 27323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:28.608438 27323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:41657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:28.608690 27323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:28.608937 27323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:28.614179 27330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.614197 27329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:28.614262 27323 server_base.cc:1034] running on GCE node
W20260812 14:27:28.614189 27332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:28.614607 27323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:28.614879 27323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:28.616034 27323 hybrid_clock.cc:648] HybridClock initialized: now 1786544848616016 us; error 32 us; skew 500 ppm
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:28.618497 27323 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:28.618916 27323 webserver.cc:460] Webserver started at http://127.23.173.193:37597/ using document root <none> and password file <none>
I20260812 14:27:28.619113 27323 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:28.619163 27323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:28.619292 27323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:28.620484 27323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "54829ec26d284cd6bf553b985a737e82"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "b397709de545e4fd7541caf90360569e"
server_key_iv: "2438c715c2c850444df2feca39d0a61a"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.620827 27323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "54829ec26d284cd6bf553b985a737e82"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "b397709de545e4fd7541caf90360569e"
server_key_iv: "2438c715c2c850444df2feca39d0a61a"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.622227 27323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:28.622946 27340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.623134 27323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:28.623227 27323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "54829ec26d284cd6bf553b985a737e82"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "b397709de545e4fd7541caf90360569e"
server_key_iv: "2438c715c2c850444df2feca39d0a61a"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.623301 27323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:28.635044 27323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:28.636950 27323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:28.637065 27323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:28.637547 27323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:28.637584 27323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.637605 27323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:28.637617 27323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.644081 27323 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:46179
I20260812 14:27:28.644235 27540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:46179 every 8 connection(s)
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:28.644536 27323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:28.645347 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27323
I20260812 14:27:28.645442 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:28.645632 24247 external_mini_cluster.cc:1428] Setting key 99bd5ab7cf6fced75f6be0d3294a7cb4
I20260812 14:27:28.647879 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.637960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.23.173.193:37439)
0812 14:27:28.638126 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.638168 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.638648 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.638910 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.638914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.639248 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:28.639528 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.639535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.640510 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.640511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.641788 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.641791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.642332 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.642333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.642471 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.642967 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:28.642977 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:28.644659 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:28.646288 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.646295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.646297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.646601 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.646768 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.646769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.646769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.646903 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.647114 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:28.647120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:28.647273 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 14:27:28.647430 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 14:27:28.647589 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260812 14:27:28.648213 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.637983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:37439 (local address 127.23.173.254:41657)
0812 14:27:28.638049 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.638051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.638254 (+ 203us) server_negotiation.cc:407] Connection header received
0812 14:27:28.638738 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.638739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.638752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.638829 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.639632 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.640413 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.641922 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.642241 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.642320 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.644754 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.644756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.644756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.644764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.646200 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.646667 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.646668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.646669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.646717 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.646958 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.646958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.646958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.647063 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.647107 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.648032 (+ 925us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.648130 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:27:28.648592 27541 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41657
I20260812 14:27:28.648675 27541 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:28.648844 27541 heartbeater.cc:511] Master 127.23.173.254:41657 requested a full tablet report, sending...
I20260812 14:27:28.649334 27203 ts_manager.cc:194] Registered new tserver with Master: 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179)
I20260812 14:27:28.650036 27203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:37439
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
I20260812 14:27:28.655609 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.650221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:41117 (local address 127.23.173.254:41657)
0812 14:27:28.650252 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.650254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.650261 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:28.650278 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.650278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.650286 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.650331 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.650711 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.651404 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.652138 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.652418 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.652494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.653285 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.653287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.653287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.653294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.654608 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.654935 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.654936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.654936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.654994 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.655170 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.655171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.655171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.655293 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.655374 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.655475 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.655537 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:27:28.655611 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.649625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.23.173.193:41117)
0812 14:27:28.649775 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.649787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.649839 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.650341 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.650342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.650544 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:28.650603 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.650605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.651509 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.651510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.652065 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.652067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.652500 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.652501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.652561 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.652955 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:28.652959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:28.653206 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:28.654704 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.654704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.654705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.654866 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.655058 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.655058 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.655059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.655121 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.655363 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:28.655364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:28.655398 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 14:27:28.655446 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:27:28.655540 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:28.672782 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41657
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:28.753955 27557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:28.754155 27557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:28.754179 27557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:28.755293 27557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:28.755334 27557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:28.755371 27557 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:27:28.755393 27557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:27:28.755422 27557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:28.756897 27557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:41657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:28.757120 27557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:28.757339 27557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:28.762341 27557 server_base.cc:1034] running on GCE node
W20260812 14:27:28.762354 27565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.762462 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.762357 27570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:28.762702 27557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:28.762933 27557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:28.764068 27557 hybrid_clock.cc:648] HybridClock initialized: now 1786544848764055 us; error 30 us; skew 500 ppm
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:28.770545 27557 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:28.770938 27557 webserver.cc:460] Webserver started at http://127.23.173.194:45867/ using document root <none> and password file <none>
I20260812 14:27:28.771135 27557 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:28.771175 27557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:28.771268 27557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:28.772329 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "52dc02bb41a84069bfd9bbaaafd005db"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "7cbd6d2b47ac4ea78e99b9721fcb8297"
server_key_iv: "6732c9e26c08c1cfce81232d7e933de1"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.772635 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "52dc02bb41a84069bfd9bbaaafd005db"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "7cbd6d2b47ac4ea78e99b9721fcb8297"
server_key_iv: "6732c9e26c08c1cfce81232d7e933de1"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.773838 27557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:28.774478 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.774631 27557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:28.774714 27557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "52dc02bb41a84069bfd9bbaaafd005db"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "7cbd6d2b47ac4ea78e99b9721fcb8297"
server_key_iv: "6732c9e26c08c1cfce81232d7e933de1"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.774775 27557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:28.783669 27557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:28.785751 27557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:28.785877 27557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:28.786363 27557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:28.786402 27557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.786423 27557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:28.786434 27557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.792869 27557 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:34577
I20260812 14:27:28.792958 27767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:34577 every 8 connection(s)
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:28.793251 27557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:28.796103 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.786811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:36779 (local address 127.23.173.254:41657)
0812 14:27:28.786881 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.786884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.787063 (+ 179us) server_negotiation.cc:407] Connection header received
0812 14:27:28.787542 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.787544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.787557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.787637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.788452 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.789199 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.790650 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.790937 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.791010 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.793636 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.793638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.793638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.793647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.794914 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.795294 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.795294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.795295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.795356 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.795545 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.795545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.795545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.795642 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.795690 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.795969 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.796038 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:27:28.796216 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.786754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.23.173.194:36779)
0812 14:27:28.786916 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.786975 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.787439 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.787692 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.787696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.788071 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:28.788346 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.788353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.789300 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.789301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.790551 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.790554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.791041 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.791042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.791188 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.791765 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:28.791775 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:28.793526 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:28.795005 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.795011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.795013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.795227 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.795392 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.795393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.795393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.795499 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.795687 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:28.795690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:28.795852 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 14:27:28.795938 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:27:28.796058 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260812 14:27:28.796669 27768 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41657
I20260812 14:27:28.796751 27768 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:28.796917 27768 heartbeater.cc:511] Master 127.23.173.254:41657 requested a full tablet report, sending...
I20260812 14:27:28.797262 27206 ts_manager.cc:194] Registered new tserver with Master: 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577)
I20260812 14:27:28.797786 27206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:36779
I20260812 14:27:28.797799 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27557
I20260812 14:27:28.797889 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:28.798094 24247 external_mini_cluster.cc:1428] Setting key 569747016d86648da4b3935835e1a8bd
I20260812 14:27:28.803120 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.797645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60035 (local address 127.23.173.254:41657)
0812 14:27:28.797683 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.797685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.797828 (+ 143us) server_negotiation.cc:407] Connection header received
0812 14:27:28.797918 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.797919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.797928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.797965 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.798329 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.798899 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.799769 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.799955 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.800016 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.800825 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.800826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.800827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.800833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.802038 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.802390 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.802391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.802391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.802441 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.802616 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.802622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.802623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.802756 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.802808 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.803000 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.803061 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:28.803174 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.797577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.23.173.194:60035)
0812 14:27:28.797765 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.797776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.797853 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.798022 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.798023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.798198 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:28.798261 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.798263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.799002 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.799002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.799652 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.799655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.800032 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.800033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.800105 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.800458 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:28.800462 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:28.800744 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:28.802118 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.802119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.802119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.802320 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.802489 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.802490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.802490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.802553 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.802824 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:28.802825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:28.802880 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 14:27:28.802967 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 14:27:28.803071 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:28.824273 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41657
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:28.905014 27786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:28.905220 27786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:28.905243 27786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:28.906374 27786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:28.906417 27786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:28.906440 27786 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:27:28.906471 27786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:27:28.906503 27786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:28.908118 27786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:41657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:28.908360 27786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:28.908618 27786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:28.913568 27786 server_base.cc:1034] running on GCE node
W20260812 14:27:28.913549 27802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.913537 27799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:28.913537 27794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:28.913949 27786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:28.914181 27786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:28.915318 27786 hybrid_clock.cc:648] HybridClock initialized: now 1786544848915305 us; error 29 us; skew 500 ppm
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:28.917714 27786 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:28.918138 27786 webserver.cc:460] Webserver started at http://127.23.173.195:45081/ using document root <none> and password file <none>
I20260812 14:27:28.918329 27786 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:28.918377 27786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:28.918500 27786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:28.919497 27786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "e27e268e5499452b893a9996616ac18f"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "874b44bdbb38f9989d4af08d1a1e6e24"
server_key_iv: "08ec7097c27b16fdbe350a8930c3367d"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.919930 27786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "e27e268e5499452b893a9996616ac18f"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "874b44bdbb38f9989d4af08d1a1e6e24"
server_key_iv: "08ec7097c27b16fdbe350a8930c3367d"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.921198 27786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:28.921885 27816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.922046 27786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:28.922137 27786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "e27e268e5499452b893a9996616ac18f"
format_stamp: "Formatted at 2026-08-12 14:27:28 on dist-test-slave-1l3l"
server_key: "874b44bdbb38f9989d4af08d1a1e6e24"
server_key_iv: "08ec7097c27b16fdbe350a8930c3367d"
server_key_version: "encryptionkey@0"
I20260812 14:27:28.922199 27786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:28.951498 27786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:28.953416 27786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:28.953559 27786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:28.954075 27786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:28.954116 27786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.954147 27786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:28.954165 27786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:28.960548 27786 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:41439
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:28.960681 28024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:41439 every 8 connection(s)
I20260812 14:27:28.960944 27786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:28.963757 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.954514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:33371 (local address 127.23.173.254:41657)
0812 14:27:28.954575 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.954578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.954771 (+ 193us) server_negotiation.cc:407] Connection header received
0812 14:27:28.955233 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.955235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.955248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.955326 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.956161 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.956906 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.958340 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.958639 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.958707 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.961230 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.961232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.961232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.961240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.962547 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.962939 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.962940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.962940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.962985 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.963190 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.963190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.963191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.963284 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.963328 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.963604 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.963663 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:27:28.963961 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.954491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.23.173.195:33371)
0812 14:27:28.954643 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.954684 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.955143 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.955413 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.955417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.955783 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:28.956058 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.956065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.956998 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.956999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.958238 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.958241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.958712 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.958713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.958859 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.959372 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:28.959381 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:28.961124 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:28.962633 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.962641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.962642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.962874 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.963044 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.963044 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.963045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.963142 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.963336 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:28.963342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:28.963497 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 14:27:28.963585 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:27:28.963740 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260812 14:27:28.964480 28025 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41657
I20260812 14:27:28.964570 28025 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:28.964742 28025 heartbeater.cc:511] Master 127.23.173.254:41657 requested a full tablet report, sending...
I20260812 14:27:28.965430 27203 ts_manager.cc:194] Registered new tserver with Master: e27e268e5499452b893a9996616ac18f (127.23.173.195:41439)
I20260812 14:27:28.965814 27203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:33371
I20260812 14:27:28.969081 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27786
I20260812 14:27:28.969148 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:28.969305 24247 external_mini_cluster.cc:1428] Setting key ad616e979112d3b2b760daa73034440e
I20260812 14:27:28.970157 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:27:28.970429 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.965455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:58255 (local address 127.23.173.254:41657)
0812 14:27:28.965503 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.965504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.965609 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:27:28.965663 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.965664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.965673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.965717 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.966021 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.966677 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.967308 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.967497 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.967539 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.968269 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.968270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.968270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.968276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.969361 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.969717 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.969718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.969718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.969777 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.969957 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.969957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.969958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.970080 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.970145 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.970312 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.970373 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:27:28.970460 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.965403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.23.173.195:58255)
0812 14:27:28.965559 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.965568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.965621 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.965719 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.965720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.965887 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:28.965946 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.965948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.966759 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.966760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.967249 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.967251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.967544 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.967544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:28.967600 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.967944 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:28.967947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:28.968200 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:28.969437 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.969438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.969439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.969660 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.969836 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:28.969836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:28.969837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:28.969903 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:28.970141 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:28.970142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:28.970188 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:27:28.970284 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 14:27:28.970377 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:27:28 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:28.992939 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.986913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.23.173.254:41657)
0812 14:27:28.986974 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.986976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.986984 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:28.987014 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.987014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.987023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.987073 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:28.987582 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.988123 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.988802 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:28.988984 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:28.989042 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:28.991035 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:28.991036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:28.991036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:28.991042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:28.992118 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.992394 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.992395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.992395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.992438 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:28.992583 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:28.992584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:28.992584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:28.992674 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:28.992745 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:28.992818 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 14:27:28.992872 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:27:28.994385 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:28.996126 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:29.002666 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.998532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:41439 (local address 127.0.0.1:55650)
0812 14:27:28.998819 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.998828 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.998876 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.999069 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.999073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.999075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:28.999146 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.999147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.000197 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.000198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.001569 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.001571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.002384 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.002385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.002454 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.002458 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.002509 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.002587 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":30,"thread_start_us":103,"threads_started":1}
I20260812 14:27:29.002697 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.998617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.23.173.194:34577)
0812 14:27:28.998919 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.998922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.998933 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:28.998975 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.998977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.998992 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.999075 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:28.999295 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.000257 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.001615 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.002275 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.002338 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.002393 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.002561 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.002598 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":72,"threads_started":1}
I20260812 14:27:29.002694 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.998648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.23.173.195:41439)
0812 14:27:28.998831 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.998836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.998926 (+ 90us) server_negotiation.cc:407] Connection header received
0812 14:27:28.998968 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.998970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.998982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:28.999047 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:28.999247 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.000121 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.001651 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.002228 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.002297 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.002351 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.002540 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.002584 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260812 14:27:29.002878 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.998470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:34577 (local address 127.0.0.1:35230)
0812 14:27:28.998696 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.998745 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.998812 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:28.999147 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:28.999148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:28.999150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:28.999217 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:28.999218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.000329 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.000330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.001524 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.001526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.002348 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.002349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.002429 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.002442 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.002515 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.002828 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":268,"thread_start_us":69,"threads_started":1}
I20260812 14:27:29.003667 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.998738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:46179 (local address 127.0.0.1:34538)
0812 14:27:28.999151 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:28.999160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:28.999203 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:29.000005 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:29.000006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:29.000007 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:29.000068 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.000069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.000971 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.000972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.001962 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.001965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.003410 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.003411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.003506 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.003511 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.003569 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.003606 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":64,"threads_started":1}
I20260812 14:27:29.003741 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:28.998956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.23.173.193:46179)
0812 14:27:28.999833 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:28.999837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:28.999848 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:28.999885 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:28.999887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:28.999900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.000139 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:29.000195 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.000891 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.002049 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.003331 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.003399 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.003447 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.003587 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.003625 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":95,"threads_started":1}
I20260812 14:27:29.003612 27661 tablet_service.cc:1467] Processing CreateTablet for tablet c6519e25e62747349745dd1401758dfa (DEFAULT_TABLE table=test-table [id=8522b2696cba472996ff5aab07c0fe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:29.003757 27911 tablet_service.cc:1467] Processing CreateTablet for tablet c6519e25e62747349745dd1401758dfa (DEFAULT_TABLE table=test-table [id=8522b2696cba472996ff5aab07c0fe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:29.004069 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6519e25e62747349745dd1401758dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.004114 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6519e25e62747349745dd1401758dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.004329 27425 tablet_service.cc:1467] Processing CreateTablet for tablet c6519e25e62747349745dd1401758dfa (DEFAULT_TABLE table=test-table [id=8522b2696cba472996ff5aab07c0fe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:29.004639 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6519e25e62747349745dd1401758dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.006872 28072 tablet_bootstrap.cc:492] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Bootstrap starting.
I20260812 14:27:29.007674 28072 tablet_bootstrap.cc:654] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.007862 28076 tablet_bootstrap.cc:492] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Bootstrap starting.
I20260812 14:27:29.008011 28072 log.cc:824] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:29.008185 28077 tablet_bootstrap.cc:492] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Bootstrap starting.
I20260812 14:27:29.008749 28077 tablet_bootstrap.cc:654] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.008828 28076 tablet_bootstrap.cc:654] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.008914 28072 tablet_bootstrap.cc:492] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: No bootstrap required, opened a new log
I20260812 14:27:29.008989 28072 ts_tablet_manager.cc:1397] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:27:29.009130 28076 log.cc:824] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:29.009195 28077 log.cc:824] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:29.009886 28077 tablet_bootstrap.cc:492] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: No bootstrap required, opened a new log
I20260812 14:27:29.009954 28077 ts_tablet_manager.cc:1397] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:29.010898 28072 raft_consensus.cc:348] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.011075 28072 raft_consensus.cc:374] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.011116 28072 raft_consensus.cc:729] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.011231 28072 consensus_queue.cc:260] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.011485 28072 ts_tablet_manager.cc:1428] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:27:29.011543 27768 heartbeater.cc:503] Master 127.23.173.254:41657 was elected leader, sending a full tablet report...
I20260812 14:27:29.011960 28077 raft_consensus.cc:348] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.012085 28077 raft_consensus.cc:374] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.012116 28077 raft_consensus.cc:729] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54829ec26d284cd6bf553b985a737e82, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.012147 28076 tablet_bootstrap.cc:492] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: No bootstrap required, opened a new log
I20260812 14:27:29.012214 28076 ts_tablet_manager.cc:1397] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 14:27:29.012243 28077 consensus_queue.cc:260] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.012523 27541 heartbeater.cc:503] Master 127.23.173.254:41657 was elected leader, sending a full tablet report...
I20260812 14:27:29.012737 28077 ts_tablet_manager.cc:1428] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:27:29.014006 28076 raft_consensus.cc:348] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.014125 28076 raft_consensus.cc:374] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.014144 28076 raft_consensus.cc:729] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27e268e5499452b893a9996616ac18f, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.014236 28076 consensus_queue.cc:260] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.014456 28076 ts_tablet_manager.cc:1428] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:27:29.014520 28025 heartbeater.cc:503] Master 127.23.173.254:41657 was elected leader, sending a full tablet report...
I20260812 14:27:29.030539 28083 raft_consensus.cc:482] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:29.030635 28083 raft_consensus.cc:504] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.030892 28083 leader_election.cc:288] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e27e268e5499452b893a9996616ac18f (127.23.173.195:41439), 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179)
I20260812 14:27:29.033681 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.031348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:47077 (local address 127.23.173.193:46179)
0812 14:27:29.031413 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:29.031415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:29.031442 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:27:29.031463 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:29.031464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:29.031472 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.031508 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:29.031652 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.032182 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.033018 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.033375 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.033439 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.033448 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.033582 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.033618 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:27:29.033687 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.031126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:59485 (local address 127.23.173.195:41439)
0812 14:27:29.031165 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:29.031167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:29.031176 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:29.031225 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:29.031225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:29.031234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.031289 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:29.031425 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.032031 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.032974 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.033354 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.033407 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.033416 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.033590 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.033625 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:29.033694 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.031067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:41439 (local address 127.23.173.194:59485)
0812 14:27:29.031118 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:29.031129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:29.031171 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:29.031293 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:29.031295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:29.031296 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:29.031365 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.031367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.032101 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.032102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.032892 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.032895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.033424 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.033425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.033525 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.033528 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.033578 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.033615 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:27:29.033703 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.031190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:46179 (local address 127.23.173.194:47077)
0812 14:27:29.031354 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:29.031365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:29.031411 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:29.031521 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:29.031521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:29.031523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:29.031598 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.031599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.032257 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.032258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.032931 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.032933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.033450 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.033451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.033520 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.033524 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.033567 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.033609 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260812 14:27:29.034219 27464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6519e25e62747349745dd1401758dfa" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82" is_pre_election: true
I20260812 14:27:29.034212 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6519e25e62747349745dd1401758dfa" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e268e5499452b893a9996616ac18f" is_pre_election: true
I20260812 14:27:29.034360 27464 raft_consensus.cc:2454] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 0.
I20260812 14:27:29.034360 27944 raft_consensus.cc:2454] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 0.
I20260812 14:27:29.034578 27583 leader_election.cc:302] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, e27e268e5499452b893a9996616ac18f; no voters:
I20260812 14:27:29.034709 28083 raft_consensus.cc:2781] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:29.034771 28083 raft_consensus.cc:482] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:29.034794 28083 raft_consensus.cc:3037] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.035483 28083 raft_consensus.cc:504] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.035594 28083 leader_election.cc:288] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 election: Requested vote from peers e27e268e5499452b893a9996616ac18f (127.23.173.195:41439), 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179)
I20260812 14:27:29.035769 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6519e25e62747349745dd1401758dfa" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e268e5499452b893a9996616ac18f"
I20260812 14:27:29.035776 27464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6519e25e62747349745dd1401758dfa" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82"
I20260812 14:27:29.035846 27944 raft_consensus.cc:3037] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.035846 27464 raft_consensus.cc:3037] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.036679 27464 raft_consensus.cc:2454] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 1.
I20260812 14:27:29.036679 27944 raft_consensus.cc:2454] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 1.
I20260812 14:27:29.036825 27583 leader_election.cc:302] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, 54829ec26d284cd6bf553b985a737e82; no voters:
I20260812 14:27:29.036919 28083 raft_consensus.cc:2781] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:29.037076 28083 raft_consensus.cc:686] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 1 LEADER]: Becoming Leader. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Running, Role: LEADER
I20260812 14:27:29.037171 28083 consensus_queue.cc:237] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } }
I20260812 14:27:29.037987 27203 catalog_manager.cc:5495] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db reported cstate change: term changed from 0 to 1, leader changed from <none> to 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194). New cstate: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } health_report { overall_health: UNKNOWN } } }
W20260812 14:27:29.043802 27769 tablet.cc:2369] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:27:29 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:29.074815 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.068337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41657 (local address 127.0.0.1:53572)
0812 14:27:29.068493 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:29.068506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:29.068584 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:29.068706 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:29.068707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:29.069016 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:29.069062 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.069065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.069705 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.069707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.070214 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.070216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.070522 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.070523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.070573 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.071037 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:29.071041 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:29.072499 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:29.073945 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:29.073952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:29.073954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:29.074162 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:29.074311 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:29.074312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:29.074312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:29.074361 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:29.074530 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:29.074534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:29.074574 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.074616 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.074719 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260812 14:27:29.074856 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.068441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.23.173.254:41657)
0812 14:27:29.068519 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:29.068522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:29.068575 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:27:29.068641 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:29.068642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:29.068651 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.068697 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:29.069148 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.069633 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.070291 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.070465 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.070518 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.072602 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:29.072603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:29.072604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:29.072612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:29.073878 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:29.074219 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:29.074220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:29.074220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:29.074267 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:29.074395 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:29.074395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:29.074395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:29.074490 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:29.074533 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.074637 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.074760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":59}
I20260812 14:27:29.076690 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:53572:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:27:29.076882 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:29.078253 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 60ef9f2821d543a2a701351e6bf599ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5de35908c806486e812e8be3fffa6acf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:27:29.078272 27911 tablet_service.cc:1467] Processing CreateTablet for tablet 60ef9f2821d543a2a701351e6bf599ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5de35908c806486e812e8be3fffa6acf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:27:29.078440 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ef9f2821d543a2a701351e6bf599ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.078440 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ef9f2821d543a2a701351e6bf599ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.078560 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 60ef9f2821d543a2a701351e6bf599ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5de35908c806486e812e8be3fffa6acf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:27:29.078729 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ef9f2821d543a2a701351e6bf599ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.079806 28076 tablet_bootstrap.cc:492] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: Bootstrap starting.
I20260812 14:27:29.079970 28077 tablet_bootstrap.cc:492] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: Bootstrap starting.
I20260812 14:27:29.080211 28072 tablet_bootstrap.cc:492] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: Bootstrap starting.
I20260812 14:27:29.080379 28076 tablet_bootstrap.cc:654] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.080610 28077 tablet_bootstrap.cc:654] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.080629 28072 tablet_bootstrap.cc:654] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.081099 28076 tablet_bootstrap.cc:492] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: No bootstrap required, opened a new log
I20260812 14:27:29.081151 28076 ts_tablet_manager.cc:1397] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:29.081264 28077 tablet_bootstrap.cc:492] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: No bootstrap required, opened a new log
I20260812 14:27:29.081298 28077 ts_tablet_manager.cc:1397] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:29.081308 28076 raft_consensus.cc:348] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.081395 28076 raft_consensus.cc:374] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.081418 28076 raft_consensus.cc:729] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27e268e5499452b893a9996616ac18f, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.081429 28077 raft_consensus.cc:348] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.081487 28077 raft_consensus.cc:374] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.081472 28076 consensus_queue.cc:260] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.081513 28077 raft_consensus.cc:729] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54829ec26d284cd6bf553b985a737e82, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.081552 28077 consensus_queue.cc:260] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.081568 28076 ts_tablet_manager.cc:1428] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:29.081627 28077 ts_tablet_manager.cc:1428] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:29.081658 28087 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.081738 28087 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:29.081779 28072 tablet_bootstrap.cc:492] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: No bootstrap required, opened a new log
I20260812 14:27:29.081771 28084 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.081820 28072 ts_tablet_manager.cc:1397] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:27:29.081832 28084 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:29.081962 28072 raft_consensus.cc:348] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.082041 28072 raft_consensus.cc:374] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.082062 28072 raft_consensus.cc:729] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.082096 28072 consensus_queue.cc:260] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.082180 28072 ts_tablet_manager.cc:1428] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:29.082217 28083 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.082312 28083 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 14:27:29.145336 27543 tablet.cc:2369] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:27:29.211810 28026 tablet.cc:2369] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:27:29.238652 28083 raft_consensus.cc:482] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:29.238767 28083 raft_consensus.cc:504] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.238932 28083 leader_election.cc:288] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), e27e268e5499452b893a9996616ac18f (127.23.173.195:41439)
I20260812 14:27:29.239209 27464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ef9f2821d543a2a701351e6bf599ad" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82" is_pre_election: true
I20260812 14:27:29.239234 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ef9f2821d543a2a701351e6bf599ad" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e268e5499452b893a9996616ac18f" is_pre_election: true
I20260812 14:27:29.239326 27944 raft_consensus.cc:2454] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 0.
I20260812 14:27:29.239326 27464 raft_consensus.cc:2454] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 0.
I20260812 14:27:29.239507 27583 leader_election.cc:302] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, 54829ec26d284cd6bf553b985a737e82; no voters:
I20260812 14:27:29.239614 28083 raft_consensus.cc:2781] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:29.239679 28083 raft_consensus.cc:482] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:29.239723 28083 raft_consensus.cc:3037] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.240432 28083 raft_consensus.cc:504] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.240546 28083 leader_election.cc:288] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 election: Requested vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), e27e268e5499452b893a9996616ac18f (127.23.173.195:41439)
I20260812 14:27:29.240722 27464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ef9f2821d543a2a701351e6bf599ad" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82"
I20260812 14:27:29.240798 27464 raft_consensus.cc:3037] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.240782 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ef9f2821d543a2a701351e6bf599ad" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e268e5499452b893a9996616ac18f"
I20260812 14:27:29.240865 27944 raft_consensus.cc:3037] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.241397 27464 raft_consensus.cc:2454] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 1.
I20260812 14:27:29.241508 27944 raft_consensus.cc:2454] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 1.
I20260812 14:27:29.241561 27583 leader_election.cc:302] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, 54829ec26d284cd6bf553b985a737e82; no voters:
I20260812 14:27:29.241665 28083 raft_consensus.cc:2781] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:29.241721 28083 raft_consensus.cc:686] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [term 1 LEADER]: Becoming Leader. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Running, Role: LEADER
I20260812 14:27:29.241763 28083 consensus_queue.cc:237] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.242105 28089 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: TxnStatusTablet state changed. Reason: New leader 52dc02bb41a84069bfd9bbaaafd005db. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.242208 28089 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:27:29.242401 27206 catalog_manager.cc:5495] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db reported cstate change: term changed from 0 to 1, leader changed from <none> to 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194). New cstate: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:29.242568 28106 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:27:29.242707 28106 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:27:29.262096 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.259379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.23.173.194:34577)
0812 14:27:29.259463 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:29.259465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:29.259490 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:27:29.259523 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:29.259524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:29.259541 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.259588 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:29.259772 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.260247 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.261348 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.261543 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.261601 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.261796 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:29.261890 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:29.261946 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.262015 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.262036 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 14:27:29.262133 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.259293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:34577 (local address 127.0.0.1:35244)
0812 14:27:29.259374 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:29.259387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:29.259439 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:29.259608 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:29.259609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:29.259610 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:29.259667 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.259670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.260319 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.260320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.261244 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.261247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.261619 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.261619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.261690 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.261708 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:29.261949 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:29.261955 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.261998 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.262049 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:27:29.264096 27464 raft_consensus.cc:1261] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Refusing update from remote peer 52dc02bb41a84069bfd9bbaaafd005db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:29.264096 27944 raft_consensus.cc:1261] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Refusing update from remote peer 52dc02bb41a84069bfd9bbaaafd005db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:29.264442 28083 consensus_queue.cc:1035] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Connected to new peer: Peer: permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:29.264587 28089 consensus_queue.cc:1035] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:29.265801 28084 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: TxnStatusTablet state changed. Reason: New leader 52dc02bb41a84069bfd9bbaaafd005db. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.265812 28087 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: TxnStatusTablet state changed. Reason: New leader 52dc02bb41a84069bfd9bbaaafd005db. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.265905 28084 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:29.265923 28087 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:29.267213 28089 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.267235 28083 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.267306 28089 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:27:29.267313 28083 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 52dc02bb41a84069bfd9bbaaafd005db: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:27:29.268172 28087 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.268179 28084 tablet_replica.cc:439] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } } }
I20260812 14:27:29.268252 28084 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P 54829ec26d284cd6bf553b985a737e82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:29.268254 28087 tablet_replica.cc:442] T 60ef9f2821d543a2a701351e6bf599ad P e27e268e5499452b893a9996616ac18f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:27:29.269253 28113 mvcc.cc:205] Tried to move back new op lower bound from 7317687702582861824 to 7317687702494633984. Current Snapshot: MvccSnapshot[applied={T|T < 7317687702582861824}]
I20260812 14:27:29.271719 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.269689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.23.173.194:34577)
0812 14:27:29.269726 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:29.269727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:29.269734 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:29.269751 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:29.269752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:29.269759 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.269810 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:29.269949 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.270403 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.271107 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.271301 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.271356 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.271484 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:29.271516 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:29.271558 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.271617 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.271637 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:29.276253 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.274206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:38023 (local address 127.23.173.194:34577)
0812 14:27:29.274249 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:29.274250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:29.274257 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:29.274273 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:29.274273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:29.274281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:29.274324 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:29.274444 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.274871 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.275564 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.275802 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.275854 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.276031 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:29.276061 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:29.276104 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:29.276161 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:27:29.276175 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:27:29.276257 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:29.274049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:34577 (local address 127.23.173.194:38023)
0812 14:27:29.274114 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:29.274124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:29.274160 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:29.274331 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:29.274331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:29.274332 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:29.274383 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.274385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.274935 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.274936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.275494 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:29.275496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:29.275856 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:29.275857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:29.275918 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:29.275934 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:29.276109 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:29.276110 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:27:29.276157 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:27:29.276192 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:27:29.278230 27464 raft_consensus.cc:1261] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Refusing update from remote peer 52dc02bb41a84069bfd9bbaaafd005db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:29.278262 27944 raft_consensus.cc:1261] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Refusing update from remote peer 52dc02bb41a84069bfd9bbaaafd005db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:29.278422 28083 consensus_queue.cc:1035] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:29.278560 28120 consensus_queue.cc:1035] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Connected to new peer: Peer: permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:29.279830 28110 mvcc.cc:205] Tried to move back new op lower bound from 7317687702642376704 to 7317687701656858624. Current Snapshot: MvccSnapshot[applied={T|T < 7317687702642376704}]
I20260812 14:27:29.280145 28112 mvcc.cc:205] Tried to move back new op lower bound from 7317687702642376704 to 7317687701656858624. Current Snapshot: MvccSnapshot[applied={T|T < 7317687702642376704}]
I20260812 14:27:29.295738 27206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:29.295841 27206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:29.296557 27206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Sending DeleteTablet for 3 replicas of tablet c6519e25e62747349745dd1401758dfa
I20260812 14:27:29.296890 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet c6519e25e62747349745dd1401758dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:29 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:55650
I20260812 14:27:29.296939 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet c6519e25e62747349745dd1401758dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:29 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:34538
I20260812 14:27:29.296940 27661 tablet_service.cc:1514] Processing DeleteTablet for tablet c6519e25e62747349745dd1401758dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:29 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:35230
I20260812 14:27:29.297300 28144 tablet_replica.cc:330] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: stopping tablet replica
I20260812 14:27:29.297266 27206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:29.297392 27206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:29.297420 28144 raft_consensus.cc:2229] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:29.297485 28144 raft_consensus.cc:2258] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:29.297636 28142 tablet_replica.cc:330] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: stopping tablet replica
I20260812 14:27:29.297734 28142 raft_consensus.cc:2229] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:29.297996 28144 ts_tablet_manager.cc:1905] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:29.298764 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9120caa0834fb386bbcef6e1aeddb7 (DEFAULT_TABLE table=test-table [id=cd1bc64e3e094ec1afa5db28cac8b6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:29.298812 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9120caa0834fb386bbcef6e1aeddb7 (DEFAULT_TABLE table=test-table [id=cd1bc64e3e094ec1afa5db28cac8b6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:29.298885 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9120caa0834fb386bbcef6e1aeddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.298923 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9120caa0834fb386bbcef6e1aeddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.299746 28144 ts_tablet_manager.cc:1918] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:29.299796 27911 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9120caa0834fb386bbcef6e1aeddb7 (DEFAULT_TABLE table=test-table [id=cd1bc64e3e094ec1afa5db28cac8b6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:29.299820 28144 log.cc:1196] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/c6519e25e62747349745dd1401758dfa
I20260812 14:27:29.299892 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9120caa0834fb386bbcef6e1aeddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:29.300069 28144 ts_tablet_manager.cc:1939] T c6519e25e62747349745dd1401758dfa P e27e268e5499452b893a9996616ac18f: Deleting consensus metadata
I20260812 14:27:29.300230 28072 tablet_bootstrap.cc:492] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Bootstrap starting.
I20260812 14:27:29.300362 28142 raft_consensus.cc:2258] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:29.300443 27178 catalog_manager.cc:4841] TS e27e268e5499452b893a9996616ac18f (127.23.173.195:41439): tablet c6519e25e62747349745dd1401758dfa (table test-table [id=8522b2696cba472996ff5aab07c0fe0e]) successfully deleted
I20260812 14:27:29.300601 28072 tablet_bootstrap.cc:654] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.300959 28076 tablet_bootstrap.cc:492] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Bootstrap starting.
I20260812 14:27:29.301295 28142 ts_tablet_manager.cc:1905] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:29.301355 28076 tablet_bootstrap.cc:654] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.301796 28072 tablet_bootstrap.cc:492] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: No bootstrap required, opened a new log
I20260812 14:27:29.301841 28072 ts_tablet_manager.cc:1397] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:27:29.301908 28077 tablet_bootstrap.cc:492] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Bootstrap starting.
I20260812 14:27:29.301977 28072 raft_consensus.cc:348] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.302054 28072 raft_consensus.cc:374] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.302084 28072 raft_consensus.cc:729] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.302125 28072 consensus_queue.cc:260] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.302219 28072 ts_tablet_manager.cc:1428] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:29.302304 28077 tablet_bootstrap.cc:654] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:29.302428 28076 tablet_bootstrap.cc:492] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: No bootstrap required, opened a new log
I20260812 14:27:29.302482 28076 ts_tablet_manager.cc:1397] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:29.302620 28076 raft_consensus.cc:348] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.302697 28076 raft_consensus.cc:374] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.302724 28076 raft_consensus.cc:729] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27e268e5499452b893a9996616ac18f, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.302760 28076 consensus_queue.cc:260] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.302862 28076 ts_tablet_manager.cc:1428] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:29.303002 28077 tablet_bootstrap.cc:492] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: No bootstrap required, opened a new log
I20260812 14:27:29.303043 28077 ts_tablet_manager.cc:1397] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:29.303166 28077 raft_consensus.cc:348] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.303216 28077 raft_consensus.cc:374] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:29.303232 28077 raft_consensus.cc:729] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54829ec26d284cd6bf553b985a737e82, State: Initialized, Role: FOLLOWER
I20260812 14:27:29.303269 28077 consensus_queue.cc:260] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.303339 28077 ts_tablet_manager.cc:1428] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:29.303565 28142 ts_tablet_manager.cc:1918] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:29.303627 28142 log.cc:1196] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/c6519e25e62747349745dd1401758dfa
I20260812 14:27:29.303844 28147 tablet_replica.cc:330] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: stopping tablet replica
I20260812 14:27:29.303893 28142 ts_tablet_manager.cc:1939] T c6519e25e62747349745dd1401758dfa P 54829ec26d284cd6bf553b985a737e82: Deleting consensus metadata
I20260812 14:27:29.303958 28147 raft_consensus.cc:2229] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:29.304072 28147 raft_consensus.cc:2258] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:29.304260 27178 catalog_manager.cc:4841] TS 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179): tablet c6519e25e62747349745dd1401758dfa (table test-table [id=8522b2696cba472996ff5aab07c0fe0e]) successfully deleted
I20260812 14:27:29.304422 28147 ts_tablet_manager.cc:1905] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:29.305644 28147 ts_tablet_manager.cc:1918] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:29.305709 28147 log.cc:1196] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/c6519e25e62747349745dd1401758dfa
I20260812 14:27:29.305936 28147 ts_tablet_manager.cc:1939] T c6519e25e62747349745dd1401758dfa P 52dc02bb41a84069bfd9bbaaafd005db: Deleting consensus metadata
I20260812 14:27:29.306341 27177 catalog_manager.cc:4841] TS 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577): tablet c6519e25e62747349745dd1401758dfa (table test-table [id=8522b2696cba472996ff5aab07c0fe0e]) successfully deleted
I20260812 14:27:29.391739 28120 raft_consensus.cc:482] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:29.391851 28120 raft_consensus.cc:504] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.392002 28120 leader_election.cc:288] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), e27e268e5499452b893a9996616ac18f (127.23.173.195:41439)
I20260812 14:27:29.392192 27464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9120caa0834fb386bbcef6e1aeddb7" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82" is_pre_election: true
I20260812 14:27:29.392202 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9120caa0834fb386bbcef6e1aeddb7" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e268e5499452b893a9996616ac18f" is_pre_election: true
I20260812 14:27:29.392280 27464 raft_consensus.cc:2454] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 0.
I20260812 14:27:29.392311 27944 raft_consensus.cc:2454] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 0.
I20260812 14:27:29.392442 27583 leader_election.cc:302] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, 54829ec26d284cd6bf553b985a737e82; no voters:
I20260812 14:27:29.392551 28120 raft_consensus.cc:2781] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:29.392604 28120 raft_consensus.cc:482] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:29.392630 28120 raft_consensus.cc:3037] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.393524 28120 raft_consensus.cc:504] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.393657 28120 leader_election.cc:288] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 election: Requested vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), e27e268e5499452b893a9996616ac18f (127.23.173.195:41439)
I20260812 14:27:29.393805 27464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9120caa0834fb386bbcef6e1aeddb7" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82"
I20260812 14:27:29.393846 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9120caa0834fb386bbcef6e1aeddb7" candidate_uuid: "52dc02bb41a84069bfd9bbaaafd005db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e268e5499452b893a9996616ac18f"
I20260812 14:27:29.393893 27464 raft_consensus.cc:3037] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.393915 27944 raft_consensus.cc:3037] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:29.394631 27944 raft_consensus.cc:2454] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 1.
I20260812 14:27:29.394631 27464 raft_consensus.cc:2454] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52dc02bb41a84069bfd9bbaaafd005db in term 1.
I20260812 14:27:29.394807 27583 leader_election.cc:302] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, 54829ec26d284cd6bf553b985a737e82; no voters:
I20260812 14:27:29.394920 28120 raft_consensus.cc:2781] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:29.395009 28120 raft_consensus.cc:686] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 1 LEADER]: Becoming Leader. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Running, Role: LEADER
I20260812 14:27:29.395064 28120 consensus_queue.cc:237] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:29.395659 27206 catalog_manager.cc:5495] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db reported cstate change: term changed from 0 to 1, leader changed from <none> to 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194). New cstate: current_term: 1 leader_uuid: "52dc02bb41a84069bfd9bbaaafd005db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:29.420372 27944 raft_consensus.cc:1261] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Refusing update from remote peer 52dc02bb41a84069bfd9bbaaafd005db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:29.420372 27464 raft_consensus.cc:1261] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Refusing update from remote peer 52dc02bb41a84069bfd9bbaaafd005db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:29.420600 28089 consensus_queue.cc:1035] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:29.420754 28083 consensus_queue.cc:1035] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [LEADER]: Connected to new peer: Peer: permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:27:29.425803 28110 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 14:27:30.945650 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:30.945761 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:30.946774 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Sending DeleteTablet for 3 replicas of tablet 5e9120caa0834fb386bbcef6e1aeddb7
I20260812 14:27:30.947079 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9120caa0834fb386bbcef6e1aeddb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:34538
I20260812 14:27:30.947069 27660 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9120caa0834fb386bbcef6e1aeddb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:35230
I20260812 14:27:30.947141 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9120caa0834fb386bbcef6e1aeddb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:55650
I20260812 14:27:30.947311 28190 tablet_replica.cc:330] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: stopping tablet replica
I20260812 14:27:30.947335 28193 tablet_replica.cc:330] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: stopping tablet replica
I20260812 14:27:30.947368 28190 raft_consensus.cc:2229] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:30.947391 28193 raft_consensus.cc:2229] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:30.947412 28190 raft_consensus.cc:2258] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:30.947477 28193 raft_consensus.cc:2258] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:30.947503 28191 tablet_replica.cc:330] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: stopping tablet replica
I20260812 14:27:30.947600 28191 raft_consensus.cc:2229] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:30.947669 28191 raft_consensus.cc:2258] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:30.947851 28193 ts_tablet_manager.cc:1905] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:30.947978 28191 ts_tablet_manager.cc:1905] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:30.949647 28193 ts_tablet_manager.cc:1918] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:27:30.949651 28191 ts_tablet_manager.cc:1918] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:27:30.949713 28191 log.cc:1196] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/5e9120caa0834fb386bbcef6e1aeddb7
I20260812 14:27:30.949816 28193 log.cc:1196] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/5e9120caa0834fb386bbcef6e1aeddb7
I20260812 14:27:30.949991 28191 ts_tablet_manager.cc:1939] T 5e9120caa0834fb386bbcef6e1aeddb7 P 52dc02bb41a84069bfd9bbaaafd005db: Deleting consensus metadata
I20260812 14:27:30.950155 28193 ts_tablet_manager.cc:1939] T 5e9120caa0834fb386bbcef6e1aeddb7 P e27e268e5499452b893a9996616ac18f: Deleting consensus metadata
I20260812 14:27:30.950340 27177 catalog_manager.cc:4841] TS 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577): tablet 5e9120caa0834fb386bbcef6e1aeddb7 (table test-table [id=cd1bc64e3e094ec1afa5db28cac8b6ec]) successfully deleted
I20260812 14:27:30.950498 27178 catalog_manager.cc:4841] TS e27e268e5499452b893a9996616ac18f (127.23.173.195:41439): tablet 5e9120caa0834fb386bbcef6e1aeddb7 (table test-table [id=cd1bc64e3e094ec1afa5db28cac8b6ec]) successfully deleted
I20260812 14:27:30.950922 28190 ts_tablet_manager.cc:1905] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:30.952414 28190 ts_tablet_manager.cc:1918] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:27:30.952477 28190 log.cc:1196] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/5e9120caa0834fb386bbcef6e1aeddb7
I20260812 14:27:30.952651 28190 ts_tablet_manager.cc:1939] T 5e9120caa0834fb386bbcef6e1aeddb7 P 54829ec26d284cd6bf553b985a737e82: Deleting consensus metadata
I20260812 14:27:30.953089 27178 catalog_manager.cc:4841] TS 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179): tablet 5e9120caa0834fb386bbcef6e1aeddb7 (table test-table [id=cd1bc64e3e094ec1afa5db28cac8b6ec]) successfully deleted
Aug 12 14:27:30 dist-test-slave-1l3l krb5kdc[27129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544850, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:27:30 dist-test-slave-1l3l krb5kdc[27129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544850, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:30.972539 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:30.965259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.23.173.254:41657)
0812 14:27:30.965428 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:30.965430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:30.965442 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:27:30.965469 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:30.965469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:30.965479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:30.965542 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:30.966112 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:30.966667 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:30.967464 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:30.967680 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:30.967795 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:30.970054 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:30.970056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:30.970056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:30.970064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:30.971486 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:30.971884 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:30.971885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:30.971885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:30.971961 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:30.972109 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:30.972109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:30.972109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:30.972213 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:30.972284 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:30.972384 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:27:30.972460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260812 14:27:30.974170 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:30.974323 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:30.975767 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec (DEFAULT_TABLE table=test-table [id=ec8ffa222b274b3eb5c65de92ff21baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:30.975790 27911 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec (DEFAULT_TABLE table=test-table [id=ec8ffa222b274b3eb5c65de92ff21baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:30.975919 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:30.975922 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:30.976967 28209 tablet_bootstrap.cc:492] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Bootstrap starting.
I20260812 14:27:30.977022 28208 tablet_bootstrap.cc:492] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Bootstrap starting.
I20260812 14:27:30.977026 27660 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec (DEFAULT_TABLE table=test-table [id=ec8ffa222b274b3eb5c65de92ff21baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:30.977152 27660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:30.977447 28209 tablet_bootstrap.cc:654] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:30.977491 28208 tablet_bootstrap.cc:654] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:30.978104 28208 tablet_bootstrap.cc:492] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: No bootstrap required, opened a new log
I20260812 14:27:30.978147 28208 ts_tablet_manager.cc:1397] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:30.978281 28208 raft_consensus.cc:348] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:30.978343 28208 raft_consensus.cc:374] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:30.978360 28208 raft_consensus.cc:729] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54829ec26d284cd6bf553b985a737e82, State: Initialized, Role: FOLLOWER
I20260812 14:27:30.978413 28208 consensus_queue.cc:260] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:30.978596 28208 ts_tablet_manager.cc:1428] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:30.978782 28216 tablet_bootstrap.cc:492] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Bootstrap starting.
I20260812 14:27:30.979156 28216 tablet_bootstrap.cc:654] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:30.979311 28209 tablet_bootstrap.cc:492] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: No bootstrap required, opened a new log
I20260812 14:27:30.979373 28209 ts_tablet_manager.cc:1397] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:30.979516 28209 raft_consensus.cc:348] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:30.979610 28209 raft_consensus.cc:374] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:30.979635 28209 raft_consensus.cc:729] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27e268e5499452b893a9996616ac18f, State: Initialized, Role: FOLLOWER
I20260812 14:27:30.979688 28209 consensus_queue.cc:260] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:30.979894 28216 tablet_bootstrap.cc:492] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: No bootstrap required, opened a new log
I20260812 14:27:30.979897 28209 ts_tablet_manager.cc:1428] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:30.979949 28216 ts_tablet_manager.cc:1397] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:30.980068 28216 raft_consensus.cc:348] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:30.980140 28216 raft_consensus.cc:374] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:30.980160 28216 raft_consensus.cc:729] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Initialized, Role: FOLLOWER
I20260812 14:27:30.980209 28216 consensus_queue.cc:260] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:30.980306 28216 ts_tablet_manager.cc:1428] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:31.006065 28218 raft_consensus.cc:482] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:31.006170 28218 raft_consensus.cc:504] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.006518 28218 leader_election.cc:288] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577)
I20260812 14:27:31.009608 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.006792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:57543 (local address 127.23.173.193:46179)
0812 14:27:31.006934 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:31.006936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:31.006991 (+ 55us) server_negotiation.cc:407] Connection header received
0812 14:27:31.007066 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:31.007067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:31.007077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:31.007138 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:31.007302 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.007871 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.008918 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.009289 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.009354 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.009364 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:31.009481 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 14:27:31.009522 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260812 14:27:31.009639 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.006768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:45695 (local address 127.23.173.194:34577)
0812 14:27:31.006925 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:31.006927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:31.007011 (+ 84us) server_negotiation.cc:407] Connection header received
0812 14:27:31.007067 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:31.007068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:31.007078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:31.007138 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:31.007322 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.007901 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.008925 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.009290 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.009355 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.009364 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:31.009534 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:27:31.009575 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260812 14:27:31.009618 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.006666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:46179 (local address 127.23.173.195:57543)
0812 14:27:31.006903 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:31.006915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:31.007003 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:31.007149 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:31.007150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:31.007152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:31.007233 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.007235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.007948 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.007949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.008828 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.008830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.009362 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.009363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.009420 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.009425 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:27:31.009466 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 14:27:31.009503 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":64,"thread_start_us":145,"threads_started":1}
I20260812 14:27:31.009893 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.006696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:34577 (local address 127.23.173.195:45695)
0812 14:27:31.006930 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:31.006941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:31.007009 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:31.007149 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:31.007150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:31.007157 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:31.007253 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.007254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.007980 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.007981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.008828 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.008831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.009391 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.009392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.009459 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.009464 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:27:31.009517 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:27:31.009555 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":7,"thread_start_us":130,"threads_started":1}
I20260812 14:27:31.009889 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab4a6c0e52a4f6db1b3084e3fe4ccec" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82" is_pre_election: true
I20260812 14:27:31.009991 27463 raft_consensus.cc:2454] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 0.
I20260812 14:27:31.010179 27820 leader_election.cc:302] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54829ec26d284cd6bf553b985a737e82, e27e268e5499452b893a9996616ac18f; no voters:
I20260812 14:27:31.010145 27692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab4a6c0e52a4f6db1b3084e3fe4ccec" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc02bb41a84069bfd9bbaaafd005db" is_pre_election: true
I20260812 14:27:31.010269 27692 raft_consensus.cc:2454] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 0.
I20260812 14:27:31.010300 28218 raft_consensus.cc:2781] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:31.010341 28218 raft_consensus.cc:482] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:31.010365 28218 raft_consensus.cc:3037] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:31.011065 28218 raft_consensus.cc:504] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.011183 28218 leader_election.cc:288] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 election: Requested vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577)
I20260812 14:27:31.011358 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab4a6c0e52a4f6db1b3084e3fe4ccec" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82"
I20260812 14:27:31.011427 27463 raft_consensus.cc:3037] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:31.011790 27692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab4a6c0e52a4f6db1b3084e3fe4ccec" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc02bb41a84069bfd9bbaaafd005db"
I20260812 14:27:31.011862 27692 raft_consensus.cc:3037] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:31.012200 27463 raft_consensus.cc:2454] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 1.
I20260812 14:27:31.012657 27692 raft_consensus.cc:2454] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 1.
I20260812 14:27:31.012888 27820 leader_election.cc:302] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54829ec26d284cd6bf553b985a737e82, e27e268e5499452b893a9996616ac18f; no voters:
I20260812 14:27:31.012984 28218 raft_consensus.cc:2781] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:31.013177 28218 raft_consensus.cc:686] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 1 LEADER]: Becoming Leader. State: Replica: e27e268e5499452b893a9996616ac18f, State: Running, Role: LEADER
I20260812 14:27:31.013264 28218 consensus_queue.cc:237] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.014123 27203 catalog_manager.cc:5495] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f reported cstate change: term changed from 0 to 1, leader changed from <none> to e27e268e5499452b893a9996616ac18f (127.23.173.195). New cstate: current_term: 1 leader_uuid: "e27e268e5499452b893a9996616ac18f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } health_report { overall_health: HEALTHY } } }
I20260812 14:27:31.032428 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.028935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.23.173.195:41439)
0812 14:27:31.029184 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:31.029186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:31.029212 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:27:31.029263 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:31.029264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:31.029290 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:31.029382 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:31.029530 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.030221 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.031363 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.031690 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.031786 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.031994 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:31.032149 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:31.032226 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:31.032278 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:27:31.032303 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":114,"threads_started":1}
I20260812 14:27:31.037958 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.035632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:49121 (local address 127.23.173.194:34577)
0812 14:27:31.035674 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:31.035676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:31.035747 (+ 71us) server_negotiation.cc:407] Connection header received
0812 14:27:31.035812 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:31.035813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:31.035821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:31.035863 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:31.035959 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.036411 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.037189 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.037386 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.037453 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.037714 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:31.037750 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:31.037804 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:31.037870 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 14:27:31.037895 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:27:31.038081 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.035504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:34577 (local address 127.23.173.195:49121)
0812 14:27:31.035673 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:31.035683 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:31.035764 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:31.035871 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:31.035871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:31.035872 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:31.035925 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.035926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.036483 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.036484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.037095 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.037097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.037508 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.037509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.037579 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.037595 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:31.037824 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:31.037826 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:27:31.037866 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:27:31.037909 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260812 14:27:31.040395 27463 raft_consensus.cc:1261] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Refusing update from remote peer e27e268e5499452b893a9996616ac18f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:31.040462 27692 raft_consensus.cc:1261] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Refusing update from remote peer e27e268e5499452b893a9996616ac18f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:31.040679 28218 consensus_queue.cc:1035] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:31.040814 28228 consensus_queue.cc:1035] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:31.048935 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.046308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:41439 (local address 127.23.173.194:53629)
0812 14:27:31.046449 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:31.046462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:31.046518 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:31.046626 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:31.046626 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:31.046628 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:27:31.046687 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.046688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.047292 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.047293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.048122 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:31.048126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.048605 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.048606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:31.048666 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.048671 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:27:31.048790 (+ 119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:27:31.048792 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:27:31.048821 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 14:27:31.048864 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20260812 14:27:31.048977 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:31.046356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:53629 (local address 127.23.173.195:41439)
0812 14:27:31.046402 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:31.046404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:31.046518 (+ 114us) server_negotiation.cc:407] Connection header received
0812 14:27:31.046574 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:31.046575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:31.046586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:31.046623 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:31.046723 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.047189 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.048210 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:31.048537 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:31.048598 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:31.048725 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:31.048754 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:31.048785 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:31.048884 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 14:27:31.048917 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:27:31.056197 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:31.056310 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:31.057034 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Sending DeleteTablet for 3 replicas of tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec
I20260812 14:27:31.057309 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:31 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:34538
I20260812 14:27:31.057422 28190 tablet_replica.cc:330] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: stopping tablet replica
I20260812 14:27:31.057410 27660 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:31 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:35230
I20260812 14:27:31.057471 28190 raft_consensus.cc:2229] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:31.057430 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:31 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:55650
I20260812 14:27:31.057497 28191 tablet_replica.cc:330] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: stopping tablet replica
I20260812 14:27:31.057538 28191 raft_consensus.cc:2229] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:31.057538 28193 tablet_replica.cc:330] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: stopping tablet replica
I20260812 14:27:31.057542 28190 raft_consensus.cc:2258] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:31.057569 28191 raft_consensus.cc:2258] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:31.057582 28193 raft_consensus.cc:2229] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:31.057674 28193 raft_consensus.cc:2258] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:31.057909 28190 ts_tablet_manager.cc:1905] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:31.057966 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:31.058127 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:31.058425 28193 ts_tablet_manager.cc:1905] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:31.059630 27425 tablet_service.cc:1467] Processing CreateTablet for tablet b1e400212c764486919a93840385ebba (DEFAULT_TABLE table=test-table [id=bba6639a29024f709f9cf34ae32ee623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:31.059860 27911 tablet_service.cc:1467] Processing CreateTablet for tablet b1e400212c764486919a93840385ebba (DEFAULT_TABLE table=test-table [id=bba6639a29024f709f9cf34ae32ee623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:31.059894 28190 ts_tablet_manager.cc:1918] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:31.059939 28190 log.cc:1196] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/1ab4a6c0e52a4f6db1b3084e3fe4ccec
I20260812 14:27:31.059964 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e400212c764486919a93840385ebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:31.060151 28190 ts_tablet_manager.cc:1939] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 54829ec26d284cd6bf553b985a737e82: Deleting consensus metadata
I20260812 14:27:31.060202 28193 ts_tablet_manager.cc:1918] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:31.060257 28193 log.cc:1196] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/1ab4a6c0e52a4f6db1b3084e3fe4ccec
I20260812 14:27:31.060458 28193 ts_tablet_manager.cc:1939] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P e27e268e5499452b893a9996616ac18f: Deleting consensus metadata
I20260812 14:27:31.060920 27178 catalog_manager.cc:4841] TS e27e268e5499452b893a9996616ac18f (127.23.173.195:41439): tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec (table test-table [id=ec8ffa222b274b3eb5c65de92ff21baa]) successfully deleted
I20260812 14:27:31.061128 28209 tablet_bootstrap.cc:492] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Bootstrap starting.
I20260812 14:27:31.061265 27178 catalog_manager.cc:4841] TS 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179): tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec (table test-table [id=ec8ffa222b274b3eb5c65de92ff21baa]) successfully deleted
I20260812 14:27:31.061389 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e400212c764486919a93840385ebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:31.061717 28209 tablet_bootstrap.cc:654] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:31.062397 28209 tablet_bootstrap.cc:492] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: No bootstrap required, opened a new log
I20260812 14:27:31.062453 28209 ts_tablet_manager.cc:1397] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:31.062661 28209 raft_consensus.cc:348] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.062721 28208 tablet_bootstrap.cc:492] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82: Bootstrap starting.
I20260812 14:27:31.062739 28209 raft_consensus.cc:374] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:31.062760 28209 raft_consensus.cc:729] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27e268e5499452b893a9996616ac18f, State: Initialized, Role: FOLLOWER
I20260812 14:27:31.062806 28209 consensus_queue.cc:260] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.062832 27660 tablet_service.cc:1467] Processing CreateTablet for tablet b1e400212c764486919a93840385ebba (DEFAULT_TABLE table=test-table [id=bba6639a29024f709f9cf34ae32ee623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:31.062917 28209 ts_tablet_manager.cc:1428] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:31.062952 27660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e400212c764486919a93840385ebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:31.063166 28208 tablet_bootstrap.cc:654] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:31.063997 28208 tablet_bootstrap.cc:492] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82: No bootstrap required, opened a new log
I20260812 14:27:31.064052 28208 ts_tablet_manager.cc:1397] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:31.064193 28208 raft_consensus.cc:348] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.064275 28208 raft_consensus.cc:374] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:31.064294 28208 raft_consensus.cc:729] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54829ec26d284cd6bf553b985a737e82, State: Initialized, Role: FOLLOWER
I20260812 14:27:31.064330 28216 tablet_bootstrap.cc:492] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Bootstrap starting.
I20260812 14:27:31.064332 28208 consensus_queue.cc:260] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.064435 28208 ts_tablet_manager.cc:1428] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:31.064668 28216 tablet_bootstrap.cc:654] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:31.064692 28191 ts_tablet_manager.cc:1905] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:31.065160 28216 tablet_bootstrap.cc:492] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: No bootstrap required, opened a new log
I20260812 14:27:31.065217 28216 ts_tablet_manager.cc:1397] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:31.065321 28216 raft_consensus.cc:348] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.065393 28216 raft_consensus.cc:374] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:31.065414 28216 raft_consensus.cc:729] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc02bb41a84069bfd9bbaaafd005db, State: Initialized, Role: FOLLOWER
I20260812 14:27:31.065459 28216 consensus_queue.cc:260] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.065578 28216 ts_tablet_manager.cc:1428] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:31.066393 28191 ts_tablet_manager.cc:1918] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:27:31.066453 28191 log.cc:1196] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/1ab4a6c0e52a4f6db1b3084e3fe4ccec
I20260812 14:27:31.066664 28191 ts_tablet_manager.cc:1939] T 1ab4a6c0e52a4f6db1b3084e3fe4ccec P 52dc02bb41a84069bfd9bbaaafd005db: Deleting consensus metadata
I20260812 14:27:31.066979 27177 catalog_manager.cc:4841] TS 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577): tablet 1ab4a6c0e52a4f6db1b3084e3fe4ccec (table test-table [id=ec8ffa222b274b3eb5c65de92ff21baa]) successfully deleted
I20260812 14:27:31.236450 28218 raft_consensus.cc:482] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:31.236578 28218 raft_consensus.cc:504] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.236740 28218 leader_election.cc:288] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577)
I20260812 14:27:31.236984 27692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e400212c764486919a93840385ebba" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc02bb41a84069bfd9bbaaafd005db" is_pre_election: true
I20260812 14:27:31.237004 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e400212c764486919a93840385ebba" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82" is_pre_election: true
I20260812 14:27:31.237079 27692 raft_consensus.cc:2454] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 0.
I20260812 14:27:31.237112 27463 raft_consensus.cc:2454] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 0.
I20260812 14:27:31.237262 27819 leader_election.cc:302] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, e27e268e5499452b893a9996616ac18f; no voters:
I20260812 14:27:31.237380 28218 raft_consensus.cc:2781] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:31.237437 28218 raft_consensus.cc:482] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:31.237452 28218 raft_consensus.cc:3037] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:31.238191 28218 raft_consensus.cc:504] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.238310 28218 leader_election.cc:288] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 election: Requested vote from peers 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179), 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577)
I20260812 14:27:31.238525 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e400212c764486919a93840385ebba" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54829ec26d284cd6bf553b985a737e82"
I20260812 14:27:31.238507 27692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e400212c764486919a93840385ebba" candidate_uuid: "e27e268e5499452b893a9996616ac18f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc02bb41a84069bfd9bbaaafd005db"
I20260812 14:27:31.238598 27463 raft_consensus.cc:3037] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:31.238598 27692 raft_consensus.cc:3037] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:31.239385 27692 raft_consensus.cc:2454] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 1.
I20260812 14:27:31.239384 27463 raft_consensus.cc:2454] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27e268e5499452b893a9996616ac18f in term 1.
I20260812 14:27:31.239567 27819 leader_election.cc:302] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc02bb41a84069bfd9bbaaafd005db, e27e268e5499452b893a9996616ac18f; no voters:
I20260812 14:27:31.239732 28218 raft_consensus.cc:2781] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:31.239800 28218 raft_consensus.cc:686] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 1 LEADER]: Becoming Leader. State: Replica: e27e268e5499452b893a9996616ac18f, State: Running, Role: LEADER
I20260812 14:27:31.239856 28218 consensus_queue.cc:237] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } }
I20260812 14:27:31.240448 27206 catalog_manager.cc:5495] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f reported cstate change: term changed from 0 to 1, leader changed from <none> to e27e268e5499452b893a9996616ac18f (127.23.173.195). New cstate: current_term: 1 leader_uuid: "e27e268e5499452b893a9996616ac18f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e27e268e5499452b893a9996616ac18f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 41439 } health_report { overall_health: HEALTHY } } }
I20260812 14:27:31.254539 27463 raft_consensus.cc:1261] T b1e400212c764486919a93840385ebba P 54829ec26d284cd6bf553b985a737e82 [term 1 FOLLOWER]: Refusing update from remote peer e27e268e5499452b893a9996616ac18f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:31.254568 27692 raft_consensus.cc:1261] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Refusing update from remote peer e27e268e5499452b893a9996616ac18f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:31.254776 28218 consensus_queue.cc:1035] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52dc02bb41a84069bfd9bbaaafd005db" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:31.254899 28228 consensus_queue.cc:1035] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54829ec26d284cd6bf553b985a737e82" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:27:31.259052 28166 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 14:27:32.776477 27202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:32.776585 27202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:32.777460 27202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc89ea4c875d40e9a8ba84f7083cd51c: Sending DeleteTablet for 3 replicas of tablet b1e400212c764486919a93840385ebba
I20260812 14:27:32.777729 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e400212c764486919a93840385ebba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:32 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:55650
I20260812 14:27:32.777907 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27323
I20260812 14:27:32.777920 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e400212c764486919a93840385ebba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:32 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:34538
I20260812 14:27:32.782748 27660 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e400212c764486919a93840385ebba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:32 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:35230
I20260812 14:27:32.784300 28301 tablet_replica.cc:330] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: stopping tablet replica
I20260812 14:27:32.784379 28301 raft_consensus.cc:2229] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:32.784426 28301 raft_consensus.cc:2258] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:32.784539 28300 tablet_replica.cc:330] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: stopping tablet replica
I20260812 14:27:32.784730 28301 ts_tablet_manager.cc:1905] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:32.786082 28300 raft_consensus.cc:2229] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:32.786186 28300 raft_consensus.cc:2258] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:32.786415 28301 ts_tablet_manager.cc:1918] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:27:32.786437 28300 ts_tablet_manager.cc:1905] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:32.786504 28301 log.cc:1196] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/b1e400212c764486919a93840385ebba
I20260812 14:27:32.786720 28301 ts_tablet_manager.cc:1939] T b1e400212c764486919a93840385ebba P 52dc02bb41a84069bfd9bbaaafd005db: Deleting consensus metadata
I20260812 14:27:32.787801 28300 ts_tablet_manager.cc:1918] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:27:32.787860 28300 log.cc:1196] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/b1e400212c764486919a93840385ebba
I20260812 14:27:32.788094 28300 ts_tablet_manager.cc:1939] T b1e400212c764486919a93840385ebba P e27e268e5499452b893a9996616ac18f: Deleting consensus metadata
I20260812 14:27:32.788414 27178 catalog_manager.cc:4841] TS e27e268e5499452b893a9996616ac18f (127.23.173.195:41439): tablet b1e400212c764486919a93840385ebba (table test-table [id=bba6639a29024f709f9cf34ae32ee623]) successfully deleted
W20260812 14:27:32.791460 27178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:46179 (error 108)
I20260812 14:27:32.791671 27177 catalog_manager.cc:4841] TS 52dc02bb41a84069bfd9bbaaafd005db (127.23.173.194:34577): tablet b1e400212c764486919a93840385ebba (table test-table [id=bba6639a29024f709f9cf34ae32ee623]) successfully deleted
I20260812 14:27:32.791903 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27557
I20260812 14:27:32.797822 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:32.791822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:46179 (local address 127.0.0.1:34546)
0812 14:27:32.791933 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:32.797774 (+ 5841us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
W20260812 14:27:32.797968 27178 catalog_manager.cc:4584] TS 54829ec26d284cd6bf553b985a737e82 (127.23.173.193:46179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1e400212c764486919a93840385ebba: Network error: Client connection negotiation failed: client connection to 127.23.173.193:46179: connect: Connection refused (error 111)
I20260812 14:27:32.803895 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27786
I20260812 14:27:32.813328 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27150
2026-08-12T14:27:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5808 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:32 dist-test-slave-1l3l krb5kdc[28313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:32 dist-test-slave-1l3l krb5kdc[28313](info): set up 2 sockets
Aug 12 14:27:32 dist-test-slave-1l3l krb5kdc[28313](info): commencing operation
krb5kdc: starting...
W20260812 14:27:33.986167 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:34.114444 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:44229
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:44229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:34.197566 28334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:34.197795 28334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:34.197822 28334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:34.199026 28334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:34.199076 28334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:34.199090 28334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:34.199101 28334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:34.199115 28334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:34.200742 28334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:44229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:44229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:34.201012 28334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:34.201242 28334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:34.206413 28334 server_base.cc:1034] running on GCE node
W20260812 14:27:34.206370 28349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.206370 28351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.206358 28348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:34.206923 28334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:34.207193 28334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:34.208345 28334 hybrid_clock.cc:648] HybridClock initialized: now 1786544854208327 us; error 33 us; skew 500 ppm
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:34.214342 28334 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:34.214840 28334 webserver.cc:460] Webserver started at http://127.23.173.254:37187/ using document root <none> and password file <none>
I20260812 14:27:34.215078 28334 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:34.215130 28334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:34.215271 28334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:34.216410 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "fd0b67f7bc8346d8bcae4b8064e00b56"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "e5d5881d9d26c68ac705db8955693c6a"
server_key_iv: "b8a96c74464bd85917689ebdaf22b413"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.216773 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "fd0b67f7bc8346d8bcae4b8064e00b56"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "e5d5881d9d26c68ac705db8955693c6a"
server_key_iv: "b8a96c74464bd85917689ebdaf22b413"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.218194 28334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:34.219064 28360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.219291 28334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:34.219412 28334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "fd0b67f7bc8346d8bcae4b8064e00b56"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "e5d5881d9d26c68ac705db8955693c6a"
server_key_iv: "b8a96c74464bd85917689ebdaf22b413"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.219498 28334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:34.242933 28334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:34.245324 28334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:34.245492 28334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:34.249753 28334 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:44229
I20260812 14:27:34.249804 28447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:44229 every 8 connection(s)
I20260812 14:27:34.250177 28334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:34.251109 28448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:34.253512 28448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56: Bootstrap starting.
I20260812 14:27:34.254109 28448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:34.254393 28448 log.cc:824] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:34.255138 28448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56: No bootstrap required, opened a new log
I20260812 14:27:34.256855 28448 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44229 } }
I20260812 14:27:34.257010 28448 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:34.257031 28448 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0b67f7bc8346d8bcae4b8064e00b56, State: Initialized, Role: FOLLOWER
I20260812 14:27:34.257167 28448 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44229 } }
I20260812 14:27:34.257238 28448 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:34.257262 28448 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:34.257313 28448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:34.257876 28448 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44229 } }
I20260812 14:27:34.257970 28448 leader_election.cc:302] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd0b67f7bc8346d8bcae4b8064e00b56; no voters:
I20260812 14:27:34.258133 28448 leader_election.cc:288] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:34.258229 28453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:34.258411 28448 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:34.258642 28453 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [term 1 LEADER]: Becoming Leader. State: Replica: fd0b67f7bc8346d8bcae4b8064e00b56, State: Running, Role: LEADER
I20260812 14:27:34.258741 28453 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44229 } }
I20260812 14:27:34.259167 28458 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0b67f7bc8346d8bcae4b8064e00b56. Latest consensus state: current_term: 1 leader_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44229 } } }
I20260812 14:27:34.259179 28456 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0b67f7bc8346d8bcae4b8064e00b56" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44229 } } }
I20260812 14:27:34.259258 28458 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:34.259291 28456 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:34.259814 28470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:34.260210 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28334
I20260812 14:27:34.260306 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:34.260366 28470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:34.260512 24247 external_mini_cluster.cc:1428] Setting key cfffa237b70ceca0ed2ff1a37f431640
I20260812 14:27:34.262238 28470 catalog_manager.cc:1347] Generated new cluster ID: 3fd07c01f4bc488aaae9180d7ab698ec
I20260812 14:27:34.262290 28470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:34.271592 28470 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:34.272205 28470 catalog_manager.cc:1504] Loading token signing keys...
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:34.277341 28470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd0b67f7bc8346d8bcae4b8064e00b56: Generated new TSK 0
I20260812 14:27:34.277514 28470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:34.278151 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.261055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.23.173.254:44229)
0812 14:27:34.261303 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:34.261313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:34.261346 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:27:34.261446 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:34.261452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:34.261497 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:34.261617 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:34.268179 (+ 6562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.269678 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.270726 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.271429 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.271537 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.274397 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:34.274402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:34.274402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:34.274416 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:34.276353 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.276754 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.276757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.276758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.276878 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.277084 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.277086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.277086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.277389 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:34.277561 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:34.277811 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 14:27:34.277969 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:34.304667 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44229
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:34.388808 28487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:34.389025 28487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:34.389048 28487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:34.390244 28487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:34.390285 28487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:34.390311 28487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:34.391767 28487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:34.391966 28487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:34.392149 28487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:34.397382 28487 server_base.cc:1034] running on GCE node
W20260812 14:27:34.397336 28499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.397339 28495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.397325 28496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:34.397858 28487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:34.398105 28487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:34.399258 28487 hybrid_clock.cc:648] HybridClock initialized: now 1786544854399235 us; error 39 us; skew 500 ppm
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:34.401818 28487 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:34.402218 28487 webserver.cc:460] Webserver started at http://127.23.173.193:37441/ using document root <none> and password file <none>
I20260812 14:27:34.402411 28487 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:34.402452 28487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:34.402585 28487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:34.403645 28487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "e3c9fdc6f3fd43789948a3956d280468"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "be5ac35542dd6e69b02907e10256a5d2"
server_key_iv: "34536107e0d5659eca135fbcc4f2bbd0"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.404014 28487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "e3c9fdc6f3fd43789948a3956d280468"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "be5ac35542dd6e69b02907e10256a5d2"
server_key_iv: "34536107e0d5659eca135fbcc4f2bbd0"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.405237 28487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:34.405929 28514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.406080 28487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:34.406158 28487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "e3c9fdc6f3fd43789948a3956d280468"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "be5ac35542dd6e69b02907e10256a5d2"
server_key_iv: "34536107e0d5659eca135fbcc4f2bbd0"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.406225 28487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:34.422061 28487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:34.424304 28487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:34.424429 28487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:34.424621 28487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:34.424924 28487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:34.424960 28487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.424981 28487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:34.425004 28487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.430845 28487 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:46669
I20260812 14:27:34.430915 28700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:46669 every 8 connection(s)
I20260812 14:27:34.431226 28487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:34.440189 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.431979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:37403 (local address 127.23.173.254:44229)
0812 14:27:34.432079 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:34.432082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:34.432258 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:27:34.432796 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:34.432798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:34.432813 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:34.432891 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:34.433815 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.434321 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.435272 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.435489 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.435544 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.437742 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:34.437744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:34.437744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:34.437751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:34.438950 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.439345 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.439346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.439346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.439406 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.439595 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.439596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.439596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.439700 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:34.439791 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:34.440045 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 14:27:34.440121 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:34.440475 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28487
I20260812 14:27:34.440549 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:34.440691 24247 external_mini_cluster.cc:1428] Setting key 9470e97f68f744439a032dcb287c8ff8
I20260812 14:27:34.440891 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.431939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44229 (local address 127.23.173.193:37403)
0812 14:27:34.432128 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:34.432171 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:34.432675 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:34.432946 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:34.432949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:34.433505 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:34.433726 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:34.433731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.434386 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.434387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:34.435196 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:34.435199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.435587 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.435588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:34.435761 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.436228 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:34.436240 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:34.437656 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:34.439037 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:34.439045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:34.439047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:34.439270 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:34.439456 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:34.439456 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:34.439456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:34.439549 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:34.439793 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:34.439800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:34.439930 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 14:27:34.440025 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 14:27:34.440160 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260812 14:27:34.441388 28702 heartbeater.cc:347] Connected to a master server at 127.23.173.254:44229
I20260812 14:27:34.441495 28702 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:34.441705 28702 heartbeater.cc:511] Master 127.23.173.254:44229 requested a full tablet report, sending...
I20260812 14:27:34.442278 28387 ts_manager.cc:194] Registered new tserver with Master: e3c9fdc6f3fd43789948a3956d280468 (127.23.173.193:46669)
I20260812 14:27:34.443090 28387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:37403
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:34.467080 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44229
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:34.548166 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:34.548367 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:34.548398 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:34.549619 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:34.549671 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:34.549706 28711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:34.551256 28711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:34.551496 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:34.551738 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:34.556782 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:34.556895 28711 server_base.cc:1034] running on GCE node
W20260812 14:27:34.556775 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.556775 28725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:34.557235 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:34.557480 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:34.558666 28711 hybrid_clock.cc:648] HybridClock initialized: now 1786544854558650 us; error 32 us; skew 500 ppm
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:34.561321 28711 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:34.561719 28711 webserver.cc:460] Webserver started at http://127.23.173.194:35293/ using document root <none> and password file <none>
I20260812 14:27:34.561936 28711 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:34.561985 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:34.562083 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:34.563120 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "25ffa4a50408494498812431924511bc"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "178873dea85563402ddb6426c0da2253"
server_key_iv: "aedfd78a6fa2295b61702c35d4e514ec"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.563421 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "25ffa4a50408494498812431924511bc"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "178873dea85563402ddb6426c0da2253"
server_key_iv: "aedfd78a6fa2295b61702c35d4e514ec"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.564720 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:34.565400 28734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.565568 28711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:34.565672 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "25ffa4a50408494498812431924511bc"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "178873dea85563402ddb6426c0da2253"
server_key_iv: "aedfd78a6fa2295b61702c35d4e514ec"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.565727 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:34.583694 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:34.585949 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:34.586071 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:34.586272 28711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:34.586534 28711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:34.586563 28711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.586596 28711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:34.586611 28711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.592415 28711 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:43577
I20260812 14:27:34.592478 28931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:43577 every 8 connection(s)
I20260812 14:27:34.592810 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:34.602175 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.593551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:37313 (local address 127.23.173.254:44229)
0812 14:27:34.593623 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:34.593626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:34.593813 (+ 187us) server_negotiation.cc:407] Connection header received
0812 14:27:34.594352 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:34.594353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:34.594365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:34.594427 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:34.595410 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.595960 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.596947 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.597159 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.597216 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.599483 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:34.599484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:34.599485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:34.599491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:34.600715 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.601113 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.601114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.601114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.601175 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.601528 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.601529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.601529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.601657 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:34.601726 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:34.602014 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 14:27:34.602106 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:27:34.602290 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.593511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44229 (local address 127.23.173.194:37313)
0812 14:27:34.593688 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:34.593727 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:34.594229 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:34.594483 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:34.594486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:34.595093 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:34.595296 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:34.595301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.596028 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.596029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:34.596856 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:34.596859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.597225 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.597225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:34.597324 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.597824 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:34.597833 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:34.599394 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:34.600813 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:34.600820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:34.600823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:34.601042 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:34.601250 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:34.601251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:34.601251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:34.601369 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:34.601733 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:34.601740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:34.601867 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:27:34.601959 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 14:27:34.602097 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260812 14:27:34.602640 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28711
I20260812 14:27:34.602743 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:34.602891 28932 heartbeater.cc:347] Connected to a master server at 127.23.173.254:44229
I20260812 14:27:34.602955 24247 external_mini_cluster.cc:1428] Setting key 3da259f4827f496a07f14e0ceaf00879
I20260812 14:27:34.602999 28932 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:34.603204 28932 heartbeater.cc:511] Master 127.23.173.254:44229 requested a full tablet report, sending...
I20260812 14:27:34.603617 28387 ts_manager.cc:194] Registered new tserver with Master: 25ffa4a50408494498812431924511bc (127.23.173.194:43577)
I20260812 14:27:34.604065 28387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:37313
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:34.630403 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44229
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:34.715139 28942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:34.715420 28942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:34.715454 28942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:34.716667 28942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:34.716714 28942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:34.716739 28942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:34.718163 28942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:34.718351 28942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:34.718544 28942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:34.723479 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.723511 28949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:34.723527 28950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:34.723795 28942 server_base.cc:1034] running on GCE node
I20260812 14:27:34.724150 28942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:34.724445 28942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:34.725610 28942 hybrid_clock.cc:648] HybridClock initialized: now 1786544854725600 us; error 33 us; skew 500 ppm
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:34.728323 28942 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:34.728812 28942 webserver.cc:460] Webserver started at http://127.23.173.195:45871/ using document root <none> and password file <none>
I20260812 14:27:34.729089 28942 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:34.729151 28942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:34.729283 28942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:34.730378 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "ba9fd44e4e0846a9be87a7607b3dbb66"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "d31052af9430536497bd885e551f1052"
server_key_iv: "0b8189fbe13d51b942f8b73f840ec720"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.730715 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "ba9fd44e4e0846a9be87a7607b3dbb66"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "d31052af9430536497bd885e551f1052"
server_key_iv: "0b8189fbe13d51b942f8b73f840ec720"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.732066 28942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:34.732851 28966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.733013 28942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:34.733096 28942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "ba9fd44e4e0846a9be87a7607b3dbb66"
format_stamp: "Formatted at 2026-08-12 14:27:34 on dist-test-slave-1l3l"
server_key: "d31052af9430536497bd885e551f1052"
server_key_iv: "0b8189fbe13d51b942f8b73f840ec720"
server_key_version: "encryptionkey@0"
I20260812 14:27:34.733163 28942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:34.746881 28942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:34.749064 28942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:34.749208 28942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:34.749442 28942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:34.749747 28942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:34.749789 28942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.749820 28942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:34.749845 28942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:34.755817 28942 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:40847
I20260812 14:27:34.755863 29170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:40847 every 8 connection(s)
I20260812 14:27:34.756212 28942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[28313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544854, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:34.765710 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.756932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60969 (local address 127.23.173.254:44229)
0812 14:27:34.757016 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:34.757018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:34.757194 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:27:34.758235 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:34.758237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:34.758250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:34.758302 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:34.758994 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.759592 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.760685 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.760898 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.760959 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.763111 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:34.763112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:34.763113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:34.763120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:34.764387 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.764796 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.764797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.764797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.764854 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:34.765083 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:34.765084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:34.765084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:34.765195 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:34.765254 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:34.765529 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 14:27:34.765616 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 14:27:34.765885 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:34.756888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44229 (local address 127.23.173.195:60969)
0812 14:27:34.757074 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:34.757116 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:34.758118 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:34.758373 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:34.758377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:34.758667 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:34.758909 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:34.758914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.759689 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.759690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:34.760598 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:34.760601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:34.760978 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:34.760979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:34.761111 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:34.761608 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:34.761616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:34.763022 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:34.764495 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:34.764504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:34.764506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:34.764726 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:34.764917 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:34.764918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:34.764918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:34.765026 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:34.765252 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:34.765259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:34.765401 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 14:27:34.765497 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 14:27:34.765662 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260812 14:27:34.766196 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28942
I20260812 14:27:34.766297 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:34.766489 29173 heartbeater.cc:347] Connected to a master server at 127.23.173.254:44229
I20260812 14:27:34.766531 24247 external_mini_cluster.cc:1428] Setting key f93a7885be1a794ebd97a2747f353a78
I20260812 14:27:34.766595 29173 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:34.766789 29173 heartbeater.cc:511] Master 127.23.173.254:44229 requested a full tablet report, sending...
I20260812 14:27:34.767166 28387 ts_manager.cc:194] Registered new tserver with Master: ba9fd44e4e0846a9be87a7607b3dbb66 (127.23.173.195:40847)
I20260812 14:27:34.767431 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:27:34.767624 28387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:60969
W20260812 14:27:34.778925 28483 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:34.778219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.23.173.254:44229)
0812 14:27:34.778259 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:34.778260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:34.778311 (+ 51us) server_negotiation.cc:407] Connection header received
0812 14:27:34.778383 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:34.778384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:34.778392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:34.778430 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:34.778856 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49080: BlockingRecv error: recv got EOF from 127.0.0.1:49080 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:34.779805 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28487
I20260812 14:27:34.804725 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28711
I20260812 14:27:34.816134 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28942
I20260812 14:27:34.822178 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28334
2026-08-12T14:27:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2010 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[29191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[29191](info): set up 2 sockets
Aug 12 14:27:34 dist-test-slave-1l3l krb5kdc[29191](info): commencing operation
krb5kdc: starting...
W20260812 14:27:35.992612 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:36 dist-test-slave-1l3l krb5kdc[29191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:36.119098 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:42249
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:36.203536 29214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:36.203791 29214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:36.203827 29214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:36.205009 29214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:36.205053 29214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:36.205070 29214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:36.205086 29214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:36.205102 29214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:36.206650 29214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:36.206895 29214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:36.207111 29214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:36.211992 29220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:36.212005 29221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:36.211982 29228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:36.212260 29214 server_base.cc:1034] running on GCE node
I20260812 14:27:36.212489 29214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:36.212749 29214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:36.213893 29214 hybrid_clock.cc:648] HybridClock initialized: now 1786544856213881 us; error 30 us; skew 500 ppm
Aug 12 14:27:36 dist-test-slave-1l3l krb5kdc[29191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544856, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:36.216295 29214 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:36.216729 29214 webserver.cc:460] Webserver started at http://127.23.173.254:41277/ using document root <none> and password file <none>
I20260812 14:27:36.216940 29214 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:36.216996 29214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:36.217121 29214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:36.218144 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "58c64b0af80a4bf39cef16a9a6aef057"
format_stamp: "Formatted at 2026-08-12 14:27:36 on dist-test-slave-1l3l"
server_key: "31f38135d69b44ba04bb7844f4dc0fc2"
server_key_iv: "cf955a51efb459caa868d903d643d58b"
server_key_version: "encryptionkey@0"
I20260812 14:27:36.218461 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "58c64b0af80a4bf39cef16a9a6aef057"
format_stamp: "Formatted at 2026-08-12 14:27:36 on dist-test-slave-1l3l"
server_key: "31f38135d69b44ba04bb7844f4dc0fc2"
server_key_iv: "cf955a51efb459caa868d903d643d58b"
server_key_version: "encryptionkey@0"
I20260812 14:27:36.219673 29214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:36.220373 29237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:36.220522 29214 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:36.220605 29214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "58c64b0af80a4bf39cef16a9a6aef057"
format_stamp: "Formatted at 2026-08-12 14:27:36 on dist-test-slave-1l3l"
server_key: "31f38135d69b44ba04bb7844f4dc0fc2"
server_key_iv: "cf955a51efb459caa868d903d643d58b"
server_key_version: "encryptionkey@0"
I20260812 14:27:36.220667 29214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:36.238243 29214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:36.240272 29214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:36.240417 29214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:36.244411 29214 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:42249
I20260812 14:27:36.244460 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:42249 every 8 connection(s)
I20260812 14:27:36.245009 29214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:36.245683 29330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:36.248005 29330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057: Bootstrap starting.
I20260812 14:27:36.248579 29330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:36.248813 29330 log.cc:824] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:36.249418 29330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057: No bootstrap required, opened a new log
I20260812 14:27:36.251005 29330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c64b0af80a4bf39cef16a9a6aef057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42249 } }
I20260812 14:27:36.251150 29330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:36.251171 29330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c64b0af80a4bf39cef16a9a6aef057, State: Initialized, Role: FOLLOWER
I20260812 14:27:36.251276 29330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c64b0af80a4bf39cef16a9a6aef057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42249 } }
I20260812 14:27:36.251328 29330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:36.251356 29330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:36.251394 29330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:36.252363 29330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c64b0af80a4bf39cef16a9a6aef057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42249 } }
I20260812 14:27:36.252478 29330 leader_election.cc:302] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c64b0af80a4bf39cef16a9a6aef057; no voters:
I20260812 14:27:36.252658 29330 leader_election.cc:288] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:36.252715 29335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:36.252877 29335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [term 1 LEADER]: Becoming Leader. State: Replica: 58c64b0af80a4bf39cef16a9a6aef057, State: Running, Role: LEADER
I20260812 14:27:36.252918 29330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:36.252977 29335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c64b0af80a4bf39cef16a9a6aef057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42249 } }
I20260812 14:27:36.253378 29337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58c64b0af80a4bf39cef16a9a6aef057. Latest consensus state: current_term: 1 leader_uuid: "58c64b0af80a4bf39cef16a9a6aef057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c64b0af80a4bf39cef16a9a6aef057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42249 } } }
I20260812 14:27:36.253476 29337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:36.253634 29336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58c64b0af80a4bf39cef16a9a6aef057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c64b0af80a4bf39cef16a9a6aef057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42249 } } }
I20260812 14:27:36.253697 29336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:36.253997 29345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:36.254537 29345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:36.254994 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29214
I20260812 14:27:36.255090 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:36.255256 24247 external_mini_cluster.cc:1428] Setting key 1bd9ab1ffcb16e902e91526edef625e8
I20260812 14:27:36.256240 29345 catalog_manager.cc:1347] Generated new cluster ID: 39d8eeb68658476aa3cd779755d48ab8
I20260812 14:27:36.256287 29345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:36 dist-test-slave-1l3l krb5kdc[29191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:36.272583 29345 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:36.273161 29345 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:36.273968 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:36.255866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.23.173.254:42249)
0812 14:27:36.256033 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:36.256044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:36.262443 (+ 6399us) server_negotiation.cc:407] Connection header received
0812 14:27:36.262617 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:36.262625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:36.262687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:36.262841 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:36.263624 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:36.265113 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:36.266193 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:36.266838 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:36.266946 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:36.269831 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:36.269837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:36.269837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:36.269850 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:36.271855 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:36.272232 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:36.272235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:36.272236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:36.272365 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:36.272574 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:36.272574 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:36.272575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:36.272947 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:36.273108 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:36.273276 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 14:27:36.273562 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260812 14:27:36.275251 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:36.278060 29345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58c64b0af80a4bf39cef16a9a6aef057: Generated new TSK 0
I20260812 14:27:36.278235 29345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:36.300586 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:36.296803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.23.173.254:42249)
0812 14:27:36.296855 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:36.296857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:36.296921 (+ 64us) server_negotiation.cc:407] Connection header received
0812 14:27:36.297005 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:36.297006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:36.297017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:36.297092 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:36.297444 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:36.298230 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:36.298963 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:36.299293 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:36.299362 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:36.300043 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:36.300045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:36.300045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:36.300305 (+ 260us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:36.300325 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:36.300333 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:36.300391 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:36.300465 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 14:27:36.300496 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:36.302995 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29214
2026-08-12T14:27:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1479 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:36 dist-test-slave-1l3l krb5kdc[29382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:36 dist-test-slave-1l3l krb5kdc[29382](info): set up 2 sockets
Aug 12 14:27:36 dist-test-slave-1l3l krb5kdc[29382](info): commencing operation
krb5kdc: starting...
W20260812 14:27:37.503993 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:37 dist-test-slave-1l3l krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:37.626838 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:45707
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:37.711215 29406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:37.711457 29406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:37.711494 29406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:37.712683 29406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:37.712729 29406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:37.712749 29406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:37.712765 29406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:37.712780 29406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:37.714316 29406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:37.714586 29406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:37.714807 29406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:37.719683 29420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:37.719683 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:37.719676 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:37.720227 29406 server_base.cc:1034] running on GCE node
I20260812 14:27:37.720470 29406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:37.720762 29406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:37.721912 29406 hybrid_clock.cc:648] HybridClock initialized: now 1786544857721893 us; error 30 us; skew 500 ppm
Aug 12 14:27:37 dist-test-slave-1l3l krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544857, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:37.724478 29406 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:37.724931 29406 webserver.cc:460] Webserver started at http://127.23.173.254:34605/ using document root <none> and password file <none>
I20260812 14:27:37.725147 29406 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:37.725198 29406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:37.725327 29406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:37.726418 29406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "186fe0993eb246f9b95475780ac3cbbc"
format_stamp: "Formatted at 2026-08-12 14:27:37 on dist-test-slave-1l3l"
server_key: "586118c0845f433f5318566628c7fe1a"
server_key_iv: "6644496586a35dfab93002e184ba76a3"
server_key_version: "encryptionkey@0"
I20260812 14:27:37.726760 29406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "186fe0993eb246f9b95475780ac3cbbc"
format_stamp: "Formatted at 2026-08-12 14:27:37 on dist-test-slave-1l3l"
server_key: "586118c0845f433f5318566628c7fe1a"
server_key_iv: "6644496586a35dfab93002e184ba76a3"
server_key_version: "encryptionkey@0"
I20260812 14:27:37.728222 29406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:37.728972 29431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:37.729152 29406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:37.729243 29406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "186fe0993eb246f9b95475780ac3cbbc"
format_stamp: "Formatted at 2026-08-12 14:27:37 on dist-test-slave-1l3l"
server_key: "586118c0845f433f5318566628c7fe1a"
server_key_iv: "6644496586a35dfab93002e184ba76a3"
server_key_version: "encryptionkey@0"
I20260812 14:27:37.729307 29406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:37.748119 29406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:37.749966 29406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:37.750114 29406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:37.754009 29406 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:45707
I20260812 14:27:37.754062 29507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:45707 every 8 connection(s)
I20260812 14:27:37.754401 29406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:37.755391 29508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:37.757748 29508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc: Bootstrap starting.
I20260812 14:27:37.758350 29508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:37.758611 29508 log.cc:824] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:37.759307 29508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc: No bootstrap required, opened a new log
I20260812 14:27:37.760957 29508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fe0993eb246f9b95475780ac3cbbc" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45707 } }
I20260812 14:27:37.761119 29508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:37.761153 29508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186fe0993eb246f9b95475780ac3cbbc, State: Initialized, Role: FOLLOWER
I20260812 14:27:37.761292 29508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fe0993eb246f9b95475780ac3cbbc" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45707 } }
I20260812 14:27:37.761353 29508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:37.761374 29508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:37.761421 29508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:37.762249 29508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fe0993eb246f9b95475780ac3cbbc" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45707 } }
I20260812 14:27:37.762349 29508 leader_election.cc:302] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 186fe0993eb246f9b95475780ac3cbbc; no voters:
I20260812 14:27:37.762538 29508 leader_election.cc:288] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:37.762575 29514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:37.762611 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29406
I20260812 14:27:37.762708 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:37.762753 29514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [term 1 LEADER]: Becoming Leader. State: Replica: 186fe0993eb246f9b95475780ac3cbbc, State: Running, Role: LEADER
I20260812 14:27:37.762825 29508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:37.762840 29514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fe0993eb246f9b95475780ac3cbbc" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45707 } }
I20260812 14:27:37.762884 24247 external_mini_cluster.cc:1428] Setting key 724b32eaae75691579327c4c02edd430
I20260812 14:27:37.763144 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 186fe0993eb246f9b95475780ac3cbbc. Latest consensus state: current_term: 1 leader_uuid: "186fe0993eb246f9b95475780ac3cbbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fe0993eb246f9b95475780ac3cbbc" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45707 } } }
I20260812 14:27:37.763214 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:37.763177 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "186fe0993eb246f9b95475780ac3cbbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fe0993eb246f9b95475780ac3cbbc" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45707 } } }
I20260812 14:27:37.763255 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:37.763804 29521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:37.764371 29521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:37.766258 29521 catalog_manager.cc:1347] Generated new cluster ID: 7e3429a1567349c99089472eabda5133
I20260812 14:27:37.766314 29521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:37 dist-test-slave-1l3l krb5kdc[29382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:37.771278 29521 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:37.771822 29521 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:37.774942 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:37.763451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.23.173.254:45707)
0812 14:27:37.763646 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:37.763656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:37.763697 (+ 41us) server_negotiation.cc:407] Connection header received
0812 14:27:37.763849 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:37.763902 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:37.763966 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:37.764957 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:37.765066 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:37.766344 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:37.767392 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:37.768038 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:37.768136 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:37.770973 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:37.770977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:37.770978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:37.770990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:37.773222 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:37.773624 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:37.773627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:37.773627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:37.773727 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:37.773917 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:37.773918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:37.773918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:37.774177 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:37.774310 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:37.774435 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 14:27:37.774699 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260812 14:27:37.775300 29521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 186fe0993eb246f9b95475780ac3cbbc: Generated new TSK 0
I20260812 14:27:37.775466 29521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:37.806877 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45707
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:37.890342 29556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:37.890554 29556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:37.890576 29556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:37.891664 29556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:37.891695 29556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:37.891748 29556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:37.893175 29556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:37.893361 29556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:37.893553 29556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:37.898327 29563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:37.898409 29556 server_base.cc:1034] running on GCE node
W20260812 14:27:37.898348 29564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:37.898372 29568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:37.898741 29556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:37.898950 29556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:37.900070 29556 hybrid_clock.cc:648] HybridClock initialized: now 1786544857900061 us; error 29 us; skew 500 ppm
Aug 12 14:27:37 dist-test-slave-1l3l krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544857, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:37.902475 29556 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:37.902927 29556 webserver.cc:460] Webserver started at http://127.23.173.193:33057/ using document root <none> and password file <none>
I20260812 14:27:37.903128 29556 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:37.903172 29556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:37.903293 29556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:37.904402 29556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "37fbc6ba3b034be9962fbd0006d67fd3"
format_stamp: "Formatted at 2026-08-12 14:27:37 on dist-test-slave-1l3l"
server_key: "e8f7987e6a02dff6414e56b1952e6070"
server_key_iv: "aa58d120840c5a93aa68edcfd00712b8"
server_key_version: "encryptionkey@0"
I20260812 14:27:37.904748 29556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "37fbc6ba3b034be9962fbd0006d67fd3"
format_stamp: "Formatted at 2026-08-12 14:27:37 on dist-test-slave-1l3l"
server_key: "e8f7987e6a02dff6414e56b1952e6070"
server_key_iv: "aa58d120840c5a93aa68edcfd00712b8"
server_key_version: "encryptionkey@0"
I20260812 14:27:37.905995 29556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:37.906687 29581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:37.906858 29556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:37.906945 29556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "37fbc6ba3b034be9962fbd0006d67fd3"
format_stamp: "Formatted at 2026-08-12 14:27:37 on dist-test-slave-1l3l"
server_key: "e8f7987e6a02dff6414e56b1952e6070"
server_key_iv: "aa58d120840c5a93aa68edcfd00712b8"
server_key_version: "encryptionkey@0"
I20260812 14:27:37.907012 29556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:37.917819 29556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:37.920022 29556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:37.920145 29556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:37.920363 29556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:37.920647 29556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:37.920681 29556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:37.920702 29556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:37.920713 29556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:37.926410 29556 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:33383
I20260812 14:27:37.926491 29772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:33383 every 8 connection(s)
I20260812 14:27:37.926792 29556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:37.932636 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29556
I20260812 14:27:37.932741 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:37.932943 24247 external_mini_cluster.cc:1428] Setting key c2ddb2544028f5dc6b647c9bbf044a5a
Aug 12 14:27:37 dist-test-slave-1l3l krb5kdc[29382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544857, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:37.936355 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:37.927563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:50471 (local address 127.23.173.254:45707)
0812 14:27:37.927621 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:37.927623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:37.927855 (+ 232us) server_negotiation.cc:407] Connection header received
0812 14:27:37.928382 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:37.928383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:37.928392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:37.928452 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:37.929422 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:37.929920 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:37.930920 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:37.931110 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:37.931152 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:37.933606 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:37.933608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:37.933608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:37.933615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:37.934975 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:37.935372 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:37.935373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:37.935373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:37.935443 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:37.935741 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:37.935742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:37.935742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:37.935858 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:37.935911 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:37.936215 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 14:27:37.936290 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:27:37.936620 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:37.927528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45707 (local address 127.23.173.193:50471)
0812 14:27:37.927730 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:37.927774 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:37.928300 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:37.928518 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:37.928522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:37.929108 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:37.929338 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:37.929343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:37.930023 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:37.930025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:37.930850 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:37.930853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:37.931185 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:37.931186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:37.931297 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:37.931821 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:37.931830 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:37.933524 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:37.935054 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:37.935061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:37.935063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:37.935282 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:37.935512 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:37.935513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:37.935514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:37.935655 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:37.935914 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:37.935920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:37.936085 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 14:27:37.936191 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 14:27:37.936361 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260812 14:27:37.937091 29773 heartbeater.cc:347] Connected to a master server at 127.23.173.254:45707
I20260812 14:27:37.937189 29773 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:37.937366 29773 heartbeater.cc:511] Master 127.23.173.254:45707 requested a full tablet report, sending...
I20260812 14:27:37.937862 29455 ts_manager.cc:194] Registered new tserver with Master: 37fbc6ba3b034be9962fbd0006d67fd3 (127.23.173.193:33383)
I20260812 14:27:37.938503 29455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:50471
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:37.959442 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45707
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:38.044525 29782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:38.044749 29782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:38.044780 29782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:38.045945 29782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:38.045990 29782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:38.046025 29782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:38.047533 29782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:38.047799 29782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:38.048028 29782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:38.052971 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:38.052976 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:38.052977 29789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:38.053177 29782 server_base.cc:1034] running on GCE node
I20260812 14:27:38.053365 29782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:38.053608 29782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:38.054757 29782 hybrid_clock.cc:648] HybridClock initialized: now 1786544858054743 us; error 30 us; skew 500 ppm
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:38.057281 29782 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:38.057694 29782 webserver.cc:460] Webserver started at http://127.23.173.194:42913/ using document root <none> and password file <none>
I20260812 14:27:38.057897 29782 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:38.057946 29782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:38.058068 29782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:38.059082 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "6896aab19bb24d0e85eabe56a610300d"
format_stamp: "Formatted at 2026-08-12 14:27:38 on dist-test-slave-1l3l"
server_key: "69fd5e3c52253d92aea147e521347a78"
server_key_iv: "ab72772c169df7f707238942ce1a31d8"
server_key_version: "encryptionkey@0"
I20260812 14:27:38.059417 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "6896aab19bb24d0e85eabe56a610300d"
format_stamp: "Formatted at 2026-08-12 14:27:38 on dist-test-slave-1l3l"
server_key: "69fd5e3c52253d92aea147e521347a78"
server_key_iv: "ab72772c169df7f707238942ce1a31d8"
server_key_version: "encryptionkey@0"
I20260812 14:27:38.060737 29782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:38.061510 29803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:38.061666 29782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:38.061750 29782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "6896aab19bb24d0e85eabe56a610300d"
format_stamp: "Formatted at 2026-08-12 14:27:38 on dist-test-slave-1l3l"
server_key: "69fd5e3c52253d92aea147e521347a78"
server_key_iv: "ab72772c169df7f707238942ce1a31d8"
server_key_version: "encryptionkey@0"
I20260812 14:27:38.061811 29782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:38.072463 29782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:38.074501 29782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:38.074635 29782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:38.074860 29782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:38.075178 29782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:38.075217 29782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:38.075248 29782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:38.075268 29782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:38.081825 29973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:45985 every 8 connection(s)
I20260812 14:27:38.082109 29782 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:45985
I20260812 14:27:38.082558 29782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:38.086535 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29782
I20260812 14:27:38.086645 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:38.086844 24247 external_mini_cluster.cc:1428] Setting key 43d77416780f17b8848b6dcf0b1e5052
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[29382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:38.093724 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:38.083390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:58389 (local address 127.23.173.254:45707)
0812 14:27:38.083464 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:38.083466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:38.083732 (+ 266us) server_negotiation.cc:407] Connection header received
0812 14:27:38.084255 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:38.084256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:38.084270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:38.084320 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:38.085524 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.086281 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.087550 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.087843 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.087905 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:38.090404 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:38.090406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:38.090406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:38.090414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:38.092051 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:38.092563 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:38.092565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:38.092565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:38.092652 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:38.092995 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:38.092996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:38.092996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:38.093152 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:38.093198 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:38.093553 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 14:27:38.093636 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:27:38.093891 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:38.083391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45707 (local address 127.23.173.194:58389)
0812 14:27:38.083567 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:38.083615 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:38.084155 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:38.084392 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:38.084396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:38.085154 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:38.085433 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:38.085439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.086385 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.086387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:38.087471 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:38.087475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.087981 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.087982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:38.088118 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:38.088660 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:38.088671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:38.090306 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:38.092136 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:38.092144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:38.092146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:38.092456 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:38.092742 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:38.092743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:38.092743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:38.092902 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:38.093247 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:38.093253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:38.093417 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 14:27:38.093513 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 14:27:38.093670 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260812 14:27:38.094425 29976 heartbeater.cc:347] Connected to a master server at 127.23.173.254:45707
I20260812 14:27:38.094534 29976 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:38.094735 29976 heartbeater.cc:511] Master 127.23.173.254:45707 requested a full tablet report, sending...
I20260812 14:27:38.095233 29455 ts_manager.cc:194] Registered new tserver with Master: 6896aab19bb24d0e85eabe56a610300d (127.23.173.194:45985)
I20260812 14:27:38.095752 29455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:58389
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:38.118835 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45707
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:38.201627 29982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:38.201829 29982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:38.201851 29982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:38.202973 29982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:38.203016 29982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:38.203043 29982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:38.204589 29982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:38.204820 29982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:38.205034 29982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:38.209996 29989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:38.210002 29990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:38.210106 29994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:38.210148 29982 server_base.cc:1034] running on GCE node
I20260812 14:27:38.210449 29982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:38.210734 29982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:38.211910 29982 hybrid_clock.cc:648] HybridClock initialized: now 1786544858211888 us; error 38 us; skew 500 ppm
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:38.214386 29982 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:38.214841 29982 webserver.cc:460] Webserver started at http://127.23.173.195:46179/ using document root <none> and password file <none>
I20260812 14:27:38.215066 29982 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:38.215111 29982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:38.215219 29982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:38.216315 29982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "37dc431f3fac4749b434f16b1d50ddc0"
format_stamp: "Formatted at 2026-08-12 14:27:38 on dist-test-slave-1l3l"
server_key: "cfdde0f9f2d742eb5bbeccf8139353c2"
server_key_iv: "57ce1d53668b3b25c0416fd6b0609077"
server_key_version: "encryptionkey@0"
I20260812 14:27:38.216648 29982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "37dc431f3fac4749b434f16b1d50ddc0"
format_stamp: "Formatted at 2026-08-12 14:27:38 on dist-test-slave-1l3l"
server_key: "cfdde0f9f2d742eb5bbeccf8139353c2"
server_key_iv: "57ce1d53668b3b25c0416fd6b0609077"
server_key_version: "encryptionkey@0"
I20260812 14:27:38.217906 29982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:38.218637 30010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:38.218780 29982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:38.218865 29982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "37dc431f3fac4749b434f16b1d50ddc0"
format_stamp: "Formatted at 2026-08-12 14:27:38 on dist-test-slave-1l3l"
server_key: "cfdde0f9f2d742eb5bbeccf8139353c2"
server_key_iv: "57ce1d53668b3b25c0416fd6b0609077"
server_key_version: "encryptionkey@0"
I20260812 14:27:38.218946 29982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:38.242269 29982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:38.244524 29982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:38.244666 29982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:38.244920 29982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:38.245272 29982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:38.245307 29982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:38.245332 29982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:38.245354 29982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:38.251451 29982 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:41337
I20260812 14:27:38.251513 30207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:41337 every 8 connection(s)
I20260812 14:27:38.251935 29982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:38.254650 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29982
I20260812 14:27:38.254762 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:38.254963 24247 external_mini_cluster.cc:1428] Setting key e5f7cad3d8fd68c17194e6d239b979e8
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[29382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:38.261746 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:38.252706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:37599 (local address 127.23.173.254:45707)
0812 14:27:38.252775 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:38.252778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:38.252970 (+ 192us) server_negotiation.cc:407] Connection header received
0812 14:27:38.253522 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:38.253523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:38.253535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:38.253602 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:38.254760 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.255462 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.256634 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.256837 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.256900 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:38.259055 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:38.259056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:38.259057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:38.259064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:38.260511 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:38.260930 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:38.260931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:38.260932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:38.260995 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:38.261215 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:38.261215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:38.261215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:38.261327 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:38.261379 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:38.261604 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 14:27:38.261676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:27:38.261940 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:38.252671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45707 (local address 127.23.173.195:37599)
0812 14:27:38.252843 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:38.252887 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:38.253403 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:38.253686 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:38.253690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:38.254348 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:38.254632 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:38.254640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.255581 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.255582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:38.256550 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:38.256554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.256906 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.256907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:38.257007 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:38.257486 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:38.257494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:38.258919 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:38.260587 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:38.260595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:38.260597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:38.260831 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:38.261059 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:38.261059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:38.261060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:38.261162 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:38.261375 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:38.261378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:38.261504 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 14:27:38.261586 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:27:38.261712 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260812 14:27:38.262470 30208 heartbeater.cc:347] Connected to a master server at 127.23.173.254:45707
I20260812 14:27:38.262571 30208 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:38.262750 30208 heartbeater.cc:511] Master 127.23.173.254:45707 requested a full tablet report, sending...
I20260812 14:27:38.263132 29455 ts_manager.cc:194] Registered new tserver with Master: 37dc431f3fac4749b434f16b1d50ddc0 (127.23.173.195:41337)
I20260812 14:27:38.263581 29455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:37599
I20260812 14:27:38.266742 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544858, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[29382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544858, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:38.291226 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:38.283512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.23.173.254:45707)
0812 14:27:38.283596 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:38.283598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:38.283649 (+ 51us) server_negotiation.cc:407] Connection header received
0812 14:27:38.283742 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:38.283744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:38.283755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:38.283811 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:38.284363 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.284847 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.285871 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:38.286078 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:38.286139 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:38.288707 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:38.288709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:38.288709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:38.288717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:38.290053 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:38.290443 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:38.290444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:38.290445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:38.290503 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:38.290653 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:38.290653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:38.290653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:38.290802 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:38.290882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:38.291066 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 14:27:38.291148 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 14:27:38.291509 29455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56420
I20260812 14:27:38.291989 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29556
I20260812 14:27:38.309901 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29782
I20260812 14:27:38.320099 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29982
I20260812 14:27:38.352559 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29406
2026-08-12T14:27:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2051 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[30231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[30231](info): set up 2 sockets
Aug 12 14:27:38 dist-test-slave-1l3l krb5kdc[30231](info): commencing operation
krb5kdc: starting...
W20260812 14:27:39.529116 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:39 dist-test-slave-1l3l krb5kdc[30231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:39.654237 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37343
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:39.736724 30252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:39.736965 30252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:39.736994 30252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:39.738149 30252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:39.738190 30252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:39.738207 30252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:39.738224 30252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:39.738238 30252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:39.739778 30252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:39.740029 30252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:39.740238 30252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:39.744988 30263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:39.744997 30268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:39.745252 30264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:39.745278 30252 server_base.cc:1034] running on GCE node
I20260812 14:27:39.745512 30252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:39.745805 30252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:39.746958 30252 hybrid_clock.cc:648] HybridClock initialized: now 1786544859746941 us; error 29 us; skew 500 ppm
Aug 12 14:27:39 dist-test-slave-1l3l krb5kdc[30231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:39.758528 30252 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:39.758983 30252 webserver.cc:460] Webserver started at http://127.23.173.254:35265/ using document root <none> and password file <none>
I20260812 14:27:39.759207 30252 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:39.759256 30252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:39.759383 30252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:39.760531 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "d5aff27085c549268c82fa27696e9eea"
format_stamp: "Formatted at 2026-08-12 14:27:39 on dist-test-slave-1l3l"
server_key: "bcb7864f85afd0c341c49afc85e3b92e"
server_key_iv: "4220bd154d454999bf9e0b52114f2b6a"
server_key_version: "encryptionkey@0"
I20260812 14:27:39.760874 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "d5aff27085c549268c82fa27696e9eea"
format_stamp: "Formatted at 2026-08-12 14:27:39 on dist-test-slave-1l3l"
server_key: "bcb7864f85afd0c341c49afc85e3b92e"
server_key_iv: "4220bd154d454999bf9e0b52114f2b6a"
server_key_version: "encryptionkey@0"
I20260812 14:27:39.762141 30252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:39.762792 30279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:39.762943 30252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:39.763027 30252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "d5aff27085c549268c82fa27696e9eea"
format_stamp: "Formatted at 2026-08-12 14:27:39 on dist-test-slave-1l3l"
server_key: "bcb7864f85afd0c341c49afc85e3b92e"
server_key_iv: "4220bd154d454999bf9e0b52114f2b6a"
server_key_version: "encryptionkey@0"
I20260812 14:27:39.763098 30252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:39.776537 30252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:39.778203 30252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:39.778339 30252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:39.782286 30252 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:37343
I20260812 14:27:39.782321 30375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:37343 every 8 connection(s)
I20260812 14:27:39.782806 30252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:39.783469 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:39.785772 30377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea: Bootstrap starting.
I20260812 14:27:39.786334 30377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:39.786568 30377 log.cc:824] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:39.787210 30377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea: No bootstrap required, opened a new log
I20260812 14:27:39.788805 30377 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5aff27085c549268c82fa27696e9eea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37343 } }
I20260812 14:27:39.788946 30377 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:39.788973 30377 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5aff27085c549268c82fa27696e9eea, State: Initialized, Role: FOLLOWER
I20260812 14:27:39.789098 30377 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5aff27085c549268c82fa27696e9eea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37343 } }
I20260812 14:27:39.789182 30377 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:39.789206 30377 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:39.789233 30377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:39.790138 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30252
I20260812 14:27:39.790153 30377 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5aff27085c549268c82fa27696e9eea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37343 } }
I20260812 14:27:39.790243 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:39.790256 30377 leader_election.cc:302] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5aff27085c549268c82fa27696e9eea; no voters:
I20260812 14:27:39.790447 24247 external_mini_cluster.cc:1428] Setting key 969dac65af85fae96beeb0d6afc99304
I20260812 14:27:39.790448 30377 leader_election.cc:288] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:39.790505 30380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:39.790657 30380 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [term 1 LEADER]: Becoming Leader. State: Replica: d5aff27085c549268c82fa27696e9eea, State: Running, Role: LEADER
I20260812 14:27:39.790733 30377 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:39.790756 30380 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5aff27085c549268c82fa27696e9eea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37343 } }
I20260812 14:27:39.791241 30382 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5aff27085c549268c82fa27696e9eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5aff27085c549268c82fa27696e9eea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37343 } } }
I20260812 14:27:39.791260 30383 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5aff27085c549268c82fa27696e9eea. Latest consensus state: current_term: 1 leader_uuid: "d5aff27085c549268c82fa27696e9eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5aff27085c549268c82fa27696e9eea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37343 } } }
I20260812 14:27:39.791337 30382 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:39.791355 30383 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:39.792325 30396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:39.792958 30396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:39.795040 30396 catalog_manager.cc:1347] Generated new cluster ID: cfb6db6266fb4a54b894a0a577077e5c
I20260812 14:27:39.795100 30396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:39 dist-test-slave-1l3l krb5kdc[30231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:39.802932 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:39.791061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.23.173.254:37343)
0812 14:27:39.791670 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:39.791682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:39.791740 (+ 58us) server_negotiation.cc:407] Connection header received
0812 14:27:39.791864 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:39.791870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:39.791914 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:39.792036 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:39.792990 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:39.794187 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:39.794973 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:39.795608 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:39.795740 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:39.798678 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:39.798684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:39.798684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:39.798703 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:39.800926 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:39.801356 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:39.801360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:39.801360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:39.801578 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:39.801760 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:39.801761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:39.801761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:39.802128 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:39.802291 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:39.802401 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 14:27:39.802689 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":77,"threads_started":1}
I20260812 14:27:39.812927 30396 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:39.813712 30396 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:39.821100 30396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5aff27085c549268c82fa27696e9eea: Generated new TSK 0
I20260812 14:27:39.821355 30396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:39.852983 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37343
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:39.937417 30422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:39.937639 30422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:39.937664 30422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:39.938896 30422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:39.938943 30422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:39.938970 30422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:39.940573 30422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:39.940812 30422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:39.941066 30422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:39.951824 30430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:39.951910 30422 server_base.cc:1034] running on GCE node
W20260812 14:27:39.951826 30431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:39.951826 30434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:39.952417 30422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:39.952698 30422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:39.953861 30422 hybrid_clock.cc:648] HybridClock initialized: now 1786544859953848 us; error 34 us; skew 500 ppm
Aug 12 14:27:39 dist-test-slave-1l3l krb5kdc[30231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:39.956543 30422 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:39.957000 30422 webserver.cc:460] Webserver started at http://127.23.173.193:44395/ using document root <none> and password file <none>
I20260812 14:27:39.957244 30422 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:39.957288 30422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:39.957386 30422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:39.958407 30422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "e0f016a3d9cb4fc9a7d87a6434897e0c"
format_stamp: "Formatted at 2026-08-12 14:27:39 on dist-test-slave-1l3l"
server_key: "0000de09c1d4afb96c938059c7d2d895"
server_key_iv: "3747dad65472d70828bfc074a8670862"
server_key_version: "encryptionkey@0"
I20260812 14:27:39.958758 30422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "e0f016a3d9cb4fc9a7d87a6434897e0c"
format_stamp: "Formatted at 2026-08-12 14:27:39 on dist-test-slave-1l3l"
server_key: "0000de09c1d4afb96c938059c7d2d895"
server_key_iv: "3747dad65472d70828bfc074a8670862"
server_key_version: "encryptionkey@0"
I20260812 14:27:39.960129 30422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:27:39.960842 30443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:39.960995 30422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:39.961091 30422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "e0f016a3d9cb4fc9a7d87a6434897e0c"
format_stamp: "Formatted at 2026-08-12 14:27:39 on dist-test-slave-1l3l"
server_key: "0000de09c1d4afb96c938059c7d2d895"
server_key_iv: "3747dad65472d70828bfc074a8670862"
server_key_version: "encryptionkey@0"
I20260812 14:27:39.961153 30422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:39.978605 30422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:39.980907 30422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:39.981069 30422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:39.981330 30422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:39.981649 30422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:39.981685 30422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:39.981715 30422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:39.981740 30422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:39.987517 30422 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:37483
I20260812 14:27:39.987568 30618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:37483 every 8 connection(s)
I20260812 14:27:39.987927 30422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:39.989100 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30422
I20260812 14:27:39.989185 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:39.989359 24247 external_mini_cluster.cc:1428] Setting key 2a2af423ebfe859346b9aa73edf8f2bf
Aug 12 14:27:39 dist-test-slave-1l3l krb5kdc[30231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:39.997939 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:39.988727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:34719 (local address 127.23.173.254:37343)
0812 14:27:39.988789 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:39.988791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:39.988956 (+ 165us) server_negotiation.cc:407] Connection header received
0812 14:27:39.990042 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:39.990060 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:39.990074 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:39.990126 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:39.990789 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:39.991512 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:39.992611 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:39.992898 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:39.992956 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:39.995181 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:39.995182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:39.995183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:39.995190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:39.996547 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:39.997046 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:39.997046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:39.997047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:39.997093 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:39.997340 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:39.997341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:39.997341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:39.997436 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:39.997479 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:39.997801 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 14:27:39.997873 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:27:39.998205 30621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:39.988677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37343 (local address 127.23.173.193:34719)
0812 14:27:39.988832 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:39.988876 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:39.989937 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:39.990208 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:39.990212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:39.990468 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:39.990698 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:39.990703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:39.991602 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:39.991603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:39.992527 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:39.992530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:39.992974 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:39.992975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:39.993108 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:39.993594 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:39.993602 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:39.995082 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:39.996646 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:39.996654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:39.996656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:39.996974 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:39.997145 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:39.997146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:39.997146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:39.997282 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:39.997490 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:39.997497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:39.997666 (+ 169us) client_negotiation.cc:769] Sending connection context
0812 14:27:39.997784 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 14:27:39.997972 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
I20260812 14:27:39.998725 30619 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37343
I20260812 14:27:39.998819 30619 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:39.999004 30619 heartbeater.cc:511] Master 127.23.173.254:37343 requested a full tablet report, sending...
I20260812 14:27:39.999503 30309 ts_manager.cc:194] Registered new tserver with Master: e0f016a3d9cb4fc9a7d87a6434897e0c (127.23.173.193:37483)
I20260812 14:27:40.000167 30309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:34719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:40.016960 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37343
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:40.100349 30625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:40.100559 30625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:40.100594 30625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:40.101756 30625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:40.101801 30625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:40.101837 30625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:40.103365 30625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:40.103616 30625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:40.103889 30625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:40.108795 30631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:40.108817 30633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:40.108958 30625 server_base.cc:1034] running on GCE node
W20260812 14:27:40.108897 30637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:40.109239 30625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:40.109488 30625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:40.110635 30625 hybrid_clock.cc:648] HybridClock initialized: now 1786544860110621 us; error 38 us; skew 500 ppm
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:40.113147 30625 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:40.113595 30625 webserver.cc:460] Webserver started at http://127.23.173.194:45963/ using document root <none> and password file <none>
I20260812 14:27:40.113812 30625 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:40.113857 30625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:40.113977 30625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:40.114997 30625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "2d337e23c09a45f4a368014b6ba6a49f"
format_stamp: "Formatted at 2026-08-12 14:27:40 on dist-test-slave-1l3l"
server_key: "e46172b13b19f9ed395197b07963e730"
server_key_iv: "4616fb936f2f097dbdb20187506b3f11"
server_key_version: "encryptionkey@0"
I20260812 14:27:40.115325 30625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "2d337e23c09a45f4a368014b6ba6a49f"
format_stamp: "Formatted at 2026-08-12 14:27:40 on dist-test-slave-1l3l"
server_key: "e46172b13b19f9ed395197b07963e730"
server_key_iv: "4616fb936f2f097dbdb20187506b3f11"
server_key_version: "encryptionkey@0"
I20260812 14:27:40.116660 30625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:40.117398 30649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.117568 30625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.117658 30625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "2d337e23c09a45f4a368014b6ba6a49f"
format_stamp: "Formatted at 2026-08-12 14:27:40 on dist-test-slave-1l3l"
server_key: "e46172b13b19f9ed395197b07963e730"
server_key_iv: "4616fb936f2f097dbdb20187506b3f11"
server_key_version: "encryptionkey@0"
I20260812 14:27:40.117713 30625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:40.133412 30625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:40.135589 30625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:40.135735 30625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:40.135982 30625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:40.136298 30625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:40.136334 30625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.136364 30625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:40.136389 30625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.141872 30625 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:43751
I20260812 14:27:40.141952 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:43751 every 8 connection(s)
I20260812 14:27:40.142256 30625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:40.142382 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30625
I20260812 14:27:40.142488 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:40.142697 24247 external_mini_cluster.cc:1428] Setting key ce4b589b1133d3c7137bbd9a5349cd1a
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
I20260812 14:27:40.153657 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:40.143002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60095 (local address 127.23.173.254:37343)
0812 14:27:40.143065 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:40.143067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:40.143341 (+ 274us) server_negotiation.cc:407] Connection header received
0812 14:27:40.144701 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:40.144702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:40.144714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:40.144764 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:40.145653 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.146425 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.147846 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.148043 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.148101 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:40.150678 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:40.150680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:40.150680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:40.150688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:40.152287 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.152753 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.152754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.152754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.152814 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.153081 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.153081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.153082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.153182 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:40.153228 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:40.153511 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 14:27:40.153586 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:27:40.153788 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:40.143009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37343 (local address 127.23.173.194:60095)
0812 14:27:40.143204 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:40.143252 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:40.144562 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:40.144837 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:40.144841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:40.145191 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:40.145524 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:40.145531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.146534 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.146536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:40.147756 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:40.147759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.148142 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.148143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:40.148331 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:40.148884 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:40.148896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:40.150566 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:40.152382 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:40.152392 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:40.152395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:40.152678 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:40.152870 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:40.152871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:40.152871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:40.153023 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:40.153229 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:40.153234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:40.153389 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 14:27:40.153474 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:27:40.153601 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260812 14:27:40.154425 30846 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37343
I20260812 14:27:40.154520 30846 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:40.154696 30846 heartbeater.cc:511] Master 127.23.173.254:37343 requested a full tablet report, sending...
I20260812 14:27:40.155131 30309 ts_manager.cc:194] Registered new tserver with Master: 2d337e23c09a45f4a368014b6ba6a49f (127.23.173.194:43751)
I20260812 14:27:40.155539 30309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:60095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:40.171175 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37343
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:40.252818 30855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:40.253029 30855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:40.253070 30855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:40.254194 30855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:40.254230 30855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:40.254263 30855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:40.255829 30855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:40.256073 30855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:40.256283 30855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:40.261181 30865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:40.261312 30855 server_base.cc:1034] running on GCE node
W20260812 14:27:40.261193 30860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:40.261235 30861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:40.261626 30855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:40.261888 30855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:40.263089 30855 hybrid_clock.cc:648] HybridClock initialized: now 1786544860263078 us; error 31 us; skew 500 ppm
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:40.265645 30855 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:40.266089 30855 webserver.cc:460] Webserver started at http://127.23.173.195:40649/ using document root <none> and password file <none>
I20260812 14:27:40.266314 30855 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:40.266359 30855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:40.266469 30855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:40.267527 30855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "4a7f2ffcfbca4e51a82d6b99f152df01"
format_stamp: "Formatted at 2026-08-12 14:27:40 on dist-test-slave-1l3l"
server_key: "cfa8c3dc25decab7e6210a1bc9ecb0d3"
server_key_iv: "bfdd69f56c004d8411fb243c66fb0e78"
server_key_version: "encryptionkey@0"
I20260812 14:27:40.267904 30855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "4a7f2ffcfbca4e51a82d6b99f152df01"
format_stamp: "Formatted at 2026-08-12 14:27:40 on dist-test-slave-1l3l"
server_key: "cfa8c3dc25decab7e6210a1bc9ecb0d3"
server_key_iv: "bfdd69f56c004d8411fb243c66fb0e78"
server_key_version: "encryptionkey@0"
I20260812 14:27:40.269191 30855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:40.269906 30874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.270087 30855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:40.270174 30855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "4a7f2ffcfbca4e51a82d6b99f152df01"
format_stamp: "Formatted at 2026-08-12 14:27:40 on dist-test-slave-1l3l"
server_key: "cfa8c3dc25decab7e6210a1bc9ecb0d3"
server_key_iv: "bfdd69f56c004d8411fb243c66fb0e78"
server_key_version: "encryptionkey@0"
I20260812 14:27:40.270247 30855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:40.280282 30855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:40.282011 30855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:40.282147 30855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:40.282349 30855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:40.282636 30855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:40.282670 30855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.282691 30855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:40.282716 30855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:40.288574 30855 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:34287
I20260812 14:27:40.288654 31056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:34287 every 8 connection(s)
I20260812 14:27:40.288938 30855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:40.296677 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30855
I20260812 14:27:40.296782 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:40.296998 24247 external_mini_cluster.cc:1428] Setting key e582e9f60ff4e09dcc0b2031e3c69af9
I20260812 14:27:40.298125 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:40.289665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:57521 (local address 127.23.173.254:37343)
0812 14:27:40.289727 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:40.289729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:40.289929 (+ 200us) server_negotiation.cc:407] Connection header received
0812 14:27:40.290848 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:40.290849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:40.290859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:40.290914 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:40.291667 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.292207 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.293190 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.293378 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.293423 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:40.295425 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:40.295426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:40.295426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:40.295434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:40.296678 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.297134 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.297135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.297135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.297199 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.297467 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.297468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.297468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.297611 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:40.297680 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:40.297971 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 14:27:40.298058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:27:40.298297 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:40.289640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37343 (local address 127.23.173.195:57521)
0812 14:27:40.289808 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:40.289850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:40.290773 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:40.290982 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:40.290986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:40.291344 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:40.291575 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:40.291580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.292274 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.292275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:40.293109 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:40.293112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.293454 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.293455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:40.293578 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:40.294000 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:40.294008 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:40.295333 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:40.296767 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:40.296775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:40.296778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:40.297059 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:40.297262 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:40.297263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:40.297263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:40.297394 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:40.297677 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:40.297684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:40.297832 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 14:27:40.297939 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 14:27:40.298098 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260812 14:27:40.298712 31057 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37343
I20260812 14:27:40.298808 31057 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:40.298980 31057 heartbeater.cc:511] Master 127.23.173.254:37343 requested a full tablet report, sending...
I20260812 14:27:40.299312 30309 ts_manager.cc:194] Registered new tserver with Master: 4a7f2ffcfbca4e51a82d6b99f152df01 (127.23.173.195:34287)
I20260812 14:27:40.299677 30309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:57521
I20260812 14:27:40.308841 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:40.333810 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:40.326061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.23.173.193:37483)
0812 14:27:40.326285 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:40.326291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:40.326303 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:27:40.326330 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:40.326332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:40.326347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:40.326421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:40.327023 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.327816 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.328608 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.328947 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.329012 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:40.331234 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:40.331243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:40.331243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:40.331250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:40.332781 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.333097 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.333100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.333100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.333169 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.333326 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.333327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.333327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.333498 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:40.333612 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:40.333653 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 14:27:40.333714 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":112,"threads_started":1}
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[30231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:40.341744 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:40.335095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.23.173.254:37343)
0812 14:27:40.335143 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:40.335145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:40.335155 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:40.335208 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:40.335209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:40.335218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:40.335261 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:40.335800 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.336293 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.336974 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:40.337174 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:40.337241 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:40.339397 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:40.339398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:40.339398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:40.339405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:40.340654 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.340994 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.340996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.340996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.341055 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:40.341278 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:40.341278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:40.341279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:40.341434 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:40.341511 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:40.341607 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:27:40.341670 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:27:40.342017 30309 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972
I20260812 14:27:40.342373 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30422
I20260812 14:27:40.355818 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30625
I20260812 14:27:40.382745 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30855
I20260812 14:27:40.389503 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30252
2026-08-12T14:27:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2037 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[31075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[31075](info): set up 2 sockets
Aug 12 14:27:40 dist-test-slave-1l3l krb5kdc[31075](info): commencing operation
krb5kdc: starting...
W20260812 14:27:41.574327 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:41 dist-test-slave-1l3l krb5kdc[31075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:41.698522 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35793
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:41.780303 31097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:41.780530 31097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:41.780551 31097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:41.781687 31097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:41.781728 31097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:41.781741 31097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:41.781752 31097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:41.781764 31097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:41.783245 31097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:41.783490 31097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:41.783732 31097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:41.788439 31103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:41.788441 31105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:41.788615 31102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:41.788908 31097 server_base.cc:1034] running on GCE node
I20260812 14:27:41.789173 31097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:41.789456 31097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:41.790607 31097 hybrid_clock.cc:648] HybridClock initialized: now 1786544861790594 us; error 32 us; skew 500 ppm
Aug 12 14:27:41 dist-test-slave-1l3l krb5kdc[31075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544861, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:41.793085 31097 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:41.793421 31097 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:41.793474 31097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:41.793601 31097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:41.794698 31097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "1358fb4f48984d28b0d15d318417b9ea"
format_stamp: "Formatted at 2026-08-12 14:27:41 on dist-test-slave-1l3l"
server_key: "28c914672705810aa8c4f77ad89ab83c"
server_key_iv: "40784dfc0e95641146726ef3b527131f"
server_key_version: "encryptionkey@0"
I20260812 14:27:41.795053 31097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "1358fb4f48984d28b0d15d318417b9ea"
format_stamp: "Formatted at 2026-08-12 14:27:41 on dist-test-slave-1l3l"
server_key: "28c914672705810aa8c4f77ad89ab83c"
server_key_iv: "40784dfc0e95641146726ef3b527131f"
server_key_version: "encryptionkey@0"
I20260812 14:27:41.796427 31097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:27:41.797117 31115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:41.797279 31097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:41.797374 31097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "1358fb4f48984d28b0d15d318417b9ea"
format_stamp: "Formatted at 2026-08-12 14:27:41 on dist-test-slave-1l3l"
server_key: "28c914672705810aa8c4f77ad89ab83c"
server_key_iv: "40784dfc0e95641146726ef3b527131f"
server_key_version: "encryptionkey@0"
I20260812 14:27:41.797435 31097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:41.808751 31097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:41.810489 31097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:41.810635 31097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:41.814595 31097 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:35793
I20260812 14:27:41.814622 31213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:35793 every 8 connection(s)
I20260812 14:27:41.814893 31097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:41.815994 31216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:41.818260 31216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea: Bootstrap starting.
I20260812 14:27:41.818840 31216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:41.819130 31216 log.cc:824] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:41.819824 31216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea: No bootstrap required, opened a new log
I20260812 14:27:41.821398 31216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1358fb4f48984d28b0d15d318417b9ea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35793 } }
I20260812 14:27:41.821539 31216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:41.821564 31216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1358fb4f48984d28b0d15d318417b9ea, State: Initialized, Role: FOLLOWER
I20260812 14:27:41.821707 31216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1358fb4f48984d28b0d15d318417b9ea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35793 } }
I20260812 14:27:41.821772 31216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:41.821795 31216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:41.821839 31216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:41.822644 31216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1358fb4f48984d28b0d15d318417b9ea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35793 } }
I20260812 14:27:41.822736 31216 leader_election.cc:302] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1358fb4f48984d28b0d15d318417b9ea; no voters:
I20260812 14:27:41.822898 31216 leader_election.cc:288] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:41.822943 31219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:41.823086 31219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [term 1 LEADER]: Becoming Leader. State: Replica: 1358fb4f48984d28b0d15d318417b9ea, State: Running, Role: LEADER
I20260812 14:27:41.823154 31216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:41.823186 31219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1358fb4f48984d28b0d15d318417b9ea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35793 } }
I20260812 14:27:41.823519 31222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1358fb4f48984d28b0d15d318417b9ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1358fb4f48984d28b0d15d318417b9ea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35793 } } }
I20260812 14:27:41.823524 31223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1358fb4f48984d28b0d15d318417b9ea. Latest consensus state: current_term: 1 leader_uuid: "1358fb4f48984d28b0d15d318417b9ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1358fb4f48984d28b0d15d318417b9ea" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35793 } } }
I20260812 14:27:41.823618 31222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:41.823623 31223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:41.823902 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 31097
I20260812 14:27:41.824031 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:41.824047 31231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:41.824234 24247 external_mini_cluster.cc:1428] Setting key 02e33e4d0d2fab2082eedd50f2b09216
I20260812 14:27:41.824555 31231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:41.826110 31231 catalog_manager.cc:1347] Generated new cluster ID: 70a40a0ebd744b60b756568bbeafb1f8
I20260812 14:27:41.826161 31231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:41 dist-test-slave-1l3l krb5kdc[31075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:41.836516 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:41.824730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.23.173.254:35793)
0812 14:27:41.824917 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:41.824927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:41.825599 (+ 672us) server_negotiation.cc:407] Connection header received
0812 14:27:41.825739 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:41.825746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:41.825805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:41.825940 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:41.826720 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:41.828135 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:41.829275 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:41.829891 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:41.829980 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:41.832842 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:41.832847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:41.832847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:41.832861 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:41.834807 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:41.835190 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:41.835193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:41.835194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:41.835316 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:41.835503 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:41.835503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:41.835504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:41.835876 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:41.836031 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:41.836126 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:27:41.836270 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260812 14:27:41.845268 31231 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:41.845837 31231 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:41.850517 31231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea: Generated new TSK 0
I20260812 14:27:41.851095 31231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:41.884310 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:35793
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:41.996702 31252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:41.996901 31252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:41.996922 31252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:41.998005 31252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:41.998039 31252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:41.998075 31252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:41.999522 31252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:41.999729 31252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:41.999934 31252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:42.004688 31259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:42.004695 31264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:42.004702 31258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:42.004802 31252 server_base.cc:1034] running on GCE node
I20260812 14:27:42.005126 31252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:42.005365 31252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:42.006507 31252 hybrid_clock.cc:648] HybridClock initialized: now 1786544862006493 us; error 30 us; skew 500 ppm
Aug 12 14:27:42 dist-test-slave-1l3l krb5kdc[31075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:42.008946 31252 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:42.009281 31252 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:42.009335 31252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:42.009459 31252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:42.010459 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "d0292807e7b84a81bd0a84621a90e4fe"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "1a7cde88ad281265ad9ac22b087060c3"
server_key_iv: "106d580d80d176eee0eff0cdeb84b128"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.010790 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "d0292807e7b84a81bd0a84621a90e4fe"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "1a7cde88ad281265ad9ac22b087060c3"
server_key_iv: "106d580d80d176eee0eff0cdeb84b128"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.012007 31252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:42.012678 31273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.012825 31252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:42.012905 31252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "d0292807e7b84a81bd0a84621a90e4fe"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "1a7cde88ad281265ad9ac22b087060c3"
server_key_iv: "106d580d80d176eee0eff0cdeb84b128"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.012965 31252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:42.028277 31252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:42.029947 31252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:42.030084 31252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:42.030288 31252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:42.030571 31252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:42.030606 31252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.030632 31252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:42.030656 31252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.036504 31252 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:40489
I20260812 14:27:42.036561 31462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:40489 every 8 connection(s)
I20260812 14:27:42.036811 31252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:42.040458 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 31252
I20260812 14:27:42.040570 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:42.040729 24247 external_mini_cluster.cc:1428] Setting key 3056f4a28702384f87b0e801225a4ae9
Aug 12 14:27:42 dist-test-slave-1l3l krb5kdc[31075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:42.046361 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.037480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:45883 (local address 127.23.173.254:35793)
0812 14:27:42.037542 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.037545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.037730 (+ 185us) server_negotiation.cc:407] Connection header received
0812 14:27:42.038242 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.038243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.038256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.038304 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:42.039381 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.039956 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.041057 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.041305 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.041366 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.043631 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:42.043632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:42.043632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:42.043641 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:42.045001 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.045395 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.045397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.045398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.045469 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.045699 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.045700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.045700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.045846 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:42.045918 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.046150 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.046263 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:27:42.046393 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.037443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35793 (local address 127.23.173.193:45883)
0812 14:27:42.037619 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.037657 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.038144 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.038386 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.038391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.038653 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:42.039279 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.039285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.040058 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.040059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.040953 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.040956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.041404 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.041405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.041539 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.042025 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:42.042033 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:42.043532 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:42.045080 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:42.045088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:42.045090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:42.045307 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:42.045541 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:42.045542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:42.045542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:42.045644 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:42.045902 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:42.045904 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:42.046022 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.046102 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.046228 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260812 14:27:42.046926 31463 heartbeater.cc:347] Connected to a master server at 127.23.173.254:35793
I20260812 14:27:42.047041 31463 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:42.047232 31463 heartbeater.cc:511] Master 127.23.173.254:35793 requested a full tablet report, sending...
I20260812 14:27:42.047804 31139 ts_manager.cc:194] Registered new tserver with Master: d0292807e7b84a81bd0a84621a90e4fe (127.23.173.193:40489)
I20260812 14:27:42.048413 31139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:45883
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:42.068179 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:35793
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:42.151444 31468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:42.151659 31468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:42.151683 31468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:42.152894 31468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:42.152940 31468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:42.152967 31468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:42.154491 31468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:42.154716 31468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:42.154933 31468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:42.160071 31478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:42.160071 31474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:42.160153 31468 server_base.cc:1034] running on GCE node
W20260812 14:27:42.160097 31475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:42.160665 31468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:42.160913 31468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:42.162053 31468 hybrid_clock.cc:648] HybridClock initialized: now 1786544862162043 us; error 31 us; skew 500 ppm
Aug 12 14:27:42 dist-test-slave-1l3l krb5kdc[31075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:42.164638 31468 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:42.164975 31468 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:42.165032 31468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:42.165158 31468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:42.166213 31468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "4f6cd470e3cc489e82dc4fda542becad"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "055bb35768cefcd5c911e1730a2bc601"
server_key_iv: "7ad298c2575b4301c6f1108329ab952b"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.166558 31468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "4f6cd470e3cc489e82dc4fda542becad"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "055bb35768cefcd5c911e1730a2bc601"
server_key_iv: "7ad298c2575b4301c6f1108329ab952b"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.167783 31468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:42.168429 31486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.168578 31468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:42.168653 31468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "4f6cd470e3cc489e82dc4fda542becad"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "055bb35768cefcd5c911e1730a2bc601"
server_key_iv: "7ad298c2575b4301c6f1108329ab952b"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.168716 31468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:42.214800 31468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:42.217135 31468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:42.217268 31468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:42.217494 31468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:42.217800 31468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:42.217836 31468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.217868 31468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:42.217891 31468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.223603 31468 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:42265
I20260812 14:27:42.223660 31674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:42265 every 8 connection(s)
I20260812 14:27:42.223944 31468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:42.224341 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 31468
I20260812 14:27:42.224475 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:42.224633 24247 external_mini_cluster.cc:1428] Setting key 2f71997d42e4d6ffe33bcb592001ec2b
Aug 12 14:27:42 dist-test-slave-1l3l krb5kdc[31075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:42.233884 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.224690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:53049 (local address 127.23.173.254:35793)
0812 14:27:42.224755 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.224758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.224968 (+ 210us) server_negotiation.cc:407] Connection header received
0812 14:27:42.225584 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.225585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.225597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.225641 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:42.226807 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.227326 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.228716 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.228988 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.229061 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.231133 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:42.231136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:42.231136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:42.231145 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:42.232629 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.232998 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.232999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.233000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.233058 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.233263 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.233264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.233265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.233408 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:42.233486 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.233703 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.233794 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:27:42.233940 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.224661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35793 (local address 127.23.173.194:53049)
0812 14:27:42.224842 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.224889 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.225482 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.225722 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.225726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.226076 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:42.226726 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.226734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.227404 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.227405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.228632 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.228635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.229063 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.229064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.229165 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.229646 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:42.229655 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:42.231036 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:42.232706 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:42.232712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:42.232714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:42.232925 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:42.233119 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:42.233120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:42.233120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:42.233210 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:42.233472 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:42.233475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:42.233585 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.233659 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.233776 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
I20260812 14:27:42.234452 31675 heartbeater.cc:347] Connected to a master server at 127.23.173.254:35793
I20260812 14:27:42.234570 31675 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:42.234716 31675 heartbeater.cc:511] Master 127.23.173.254:35793 requested a full tablet report, sending...
I20260812 14:27:42.235046 31139 ts_manager.cc:194] Registered new tserver with Master: 4f6cd470e3cc489e82dc4fda542becad (127.23.173.194:42265)
I20260812 14:27:42.235486 31139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:53049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:42.251950 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:35793
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:42.333334 31684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:42.333534 31684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:42.333557 31684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:42.334645 31684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:42.334677 31684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:42.334702 31684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:42.336197 31684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:42.336406 31684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:42.336627 31684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:42.341501 31684 server_base.cc:1034] running on GCE node
W20260812 14:27:42.341423 31697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:42.341444 31695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:42.341567 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:42.341861 31684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:42.342096 31684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:42.343242 31684 hybrid_clock.cc:648] HybridClock initialized: now 1786544862343223 us; error 42 us; skew 500 ppm
Aug 12 14:27:42 dist-test-slave-1l3l krb5kdc[31075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:42.345693 31684 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:42.345996 31684 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:42.346047 31684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:42.346174 31684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:42.347191 31684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "192c0a30e9504498b475201d94400ed9"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "6c7a628a7c19b821f6b7b8ad78072622"
server_key_iv: "97e70da4e444c89afe46e269f986b2e3"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.347523 31684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "192c0a30e9504498b475201d94400ed9"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "6c7a628a7c19b821f6b7b8ad78072622"
server_key_iv: "97e70da4e444c89afe46e269f986b2e3"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.348731 31684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:42.349393 31712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.349541 31684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:42.349637 31684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "192c0a30e9504498b475201d94400ed9"
format_stamp: "Formatted at 2026-08-12 14:27:42 on dist-test-slave-1l3l"
server_key: "6c7a628a7c19b821f6b7b8ad78072622"
server_key_iv: "97e70da4e444c89afe46e269f986b2e3"
server_key_version: "encryptionkey@0"
I20260812 14:27:42.349699 31684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:42.360781 31684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:42.362577 31684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:42.362707 31684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:42.362915 31684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:42.363204 31684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:42.363243 31684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.363272 31684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:42.363301 31684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:42.368940 31684 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:36189
I20260812 14:27:42.368989 31905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:36189 every 8 connection(s)
I20260812 14:27:42.369230 31684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:27:42 dist-test-slave-1l3l krb5kdc[31075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:42.376994 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 31684
I20260812 14:27:42.377074 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:42.377269 24247 external_mini_cluster.cc:1428] Setting key 465048a05633920bdc9d9287522d0c08
I20260812 14:27:42.377785 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.369876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:35771 (local address 127.23.173.254:35793)
0812 14:27:42.369930 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.369932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.370129 (+ 197us) server_negotiation.cc:407] Connection header received
0812 14:27:42.370575 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.370576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.370585 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.370639 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:42.371576 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.372121 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.373076 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.373262 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.373316 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.375258 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:42.375259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:42.375259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:42.375266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:42.376398 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.376765 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.376766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.376767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.376863 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.377103 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.377104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.377104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.377245 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:42.377319 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.377605 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.377691 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:27:42.377952 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.369849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35793 (local address 127.23.173.195:35771)
0812 14:27:42.369999 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.370040 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.370499 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.370703 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.370706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.370960 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:42.371492 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.371498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.372186 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.372187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.373000 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.373002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.373323 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.373324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.373418 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.373812 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:42.373819 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:42.375168 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:42.376461 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:42.376467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:42.376469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:42.376678 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:42.376945 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:42.376946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:42.376946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:42.377035 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:42.377308 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:42.377315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:42.377460 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.377569 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.377729 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260812 14:27:42.378469 31907 heartbeater.cc:347] Connected to a master server at 127.23.173.254:35793
I20260812 14:27:42.378559 31907 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:42.378681 31907 heartbeater.cc:511] Master 127.23.173.254:35793 requested a full tablet report, sending...
I20260812 14:27:42.378998 31139 ts_manager.cc:194] Registered new tserver with Master: 192c0a30e9504498b475201d94400ed9 (127.23.173.195:36189)
I20260812 14:27:42.379328 31139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:35771
I20260812 14:27:42.389127 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:27:42.395846 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.390447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.23.173.254:35793)
0812 14:27:42.390506 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.390508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.390575 (+ 67us) server_negotiation.cc:407] Connection header received
0812 14:27:42.390643 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.390644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.390653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.390695 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:42.391237 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.391779 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.392460 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.392663 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.392717 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.393745 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:42.393746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:42.393746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:42.393754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:42.394755 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.395067 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.395069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.395069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.395124 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:42.395278 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:42.395279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:42.395279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:42.395438 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:42.395533 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.395650 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.395756 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:42.397145 31139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:42.398665 31139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:42.404897 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.400903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:40489 (local address 127.0.0.1:33828)
0812 14:27:42.401282 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.401292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.401337 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.401610 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.401611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.401613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:42.401688 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.401690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.402636 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.402637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.403796 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.403799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.404605 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.404605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.404694 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.404708 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.404771 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.404822 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":91,"threads_started":1}
I20260812 14:27:42.404909 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.401175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.23.173.193:40489)
0812 14:27:42.401346 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.401351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.401361 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:42.401403 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.401405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.401419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.401513 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:42.401772 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.402551 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.403883 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.404522 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.404587 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.404647 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.404794 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.404833 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260812 14:27:42.405308 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.400887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:42265 (local address 127.0.0.1:44470)
0812 14:27:42.401048 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.401119 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.401186 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.401599 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.401602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.401604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:42.401674 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.401676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.403020 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.403021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.404042 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.404044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.404936 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.404937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.405168 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.405172 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.405202 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.405239 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260812 14:27:42.405426 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.400913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:36189 (local address 127.0.0.1:60534)
0812 14:27:42.401359 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.401370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.401424 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.401903 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.401903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.401905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:42.401937 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.401939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.403174 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.403175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.404149 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.404152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.405162 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.405163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.405222 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.405226 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.405280 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.405318 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":81,"threads_started":1}
I20260812 14:27:42.405483 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.401161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.23.173.195:36189)
0812 14:27:42.401674 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.401678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.401688 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:42.401728 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.401729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.401745 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.401814 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:42.402022 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.403084 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.404231 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.404907 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.405138 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.405204 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.405364 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.405411 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":106,"threads_started":1}
I20260812 14:27:42.405656 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.401097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.23.173.194:42265)
0812 14:27:42.401394 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.401399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.401409 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:42.401449 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.401451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.401466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.401544 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:42.401840 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.402912 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.404114 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.404830 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.405428 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.405504 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.405533 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.405577 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20260812 14:27:42.405715 31349 tablet_service.cc:1467] Processing CreateTablet for tablet 11bd614406d343a39be8a75543f8685b (DEFAULT_TABLE table=test-table [id=7bbd62869edf4cd7b46dfa8fb0fe4b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:42.406100 31349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bd614406d343a39be8a75543f8685b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:42.406117 31796 tablet_service.cc:1467] Processing CreateTablet for tablet 11bd614406d343a39be8a75543f8685b (DEFAULT_TABLE table=test-table [id=7bbd62869edf4cd7b46dfa8fb0fe4b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:42.406366 31796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bd614406d343a39be8a75543f8685b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:42.408615 31940 tablet_bootstrap.cc:492] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: Bootstrap starting.
I20260812 14:27:42.408854 31942 tablet_bootstrap.cc:492] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Bootstrap starting.
I20260812 14:27:42.409283 31940 tablet_bootstrap.cc:654] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:42.409348 31942 tablet_bootstrap.cc:654] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:42.409319 31570 tablet_service.cc:1467] Processing CreateTablet for tablet 11bd614406d343a39be8a75543f8685b (DEFAULT_TABLE table=test-table [id=7bbd62869edf4cd7b46dfa8fb0fe4b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:42.409641 31570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bd614406d343a39be8a75543f8685b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:42.409855 31942 log.cc:824] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:42.409909 31940 log.cc:824] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:42.410575 31940 tablet_bootstrap.cc:492] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: No bootstrap required, opened a new log
I20260812 14:27:42.410625 31940 ts_tablet_manager.cc:1397] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:42.410653 31942 tablet_bootstrap.cc:492] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: No bootstrap required, opened a new log
I20260812 14:27:42.410717 31942 ts_tablet_manager.cc:1397] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:42.412127 31940 raft_consensus.cc:348] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.412284 31940 raft_consensus.cc:374] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:42.412312 31940 raft_consensus.cc:729] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0292807e7b84a81bd0a84621a90e4fe, State: Initialized, Role: FOLLOWER
I20260812 14:27:42.412424 31940 consensus_queue.cc:260] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.412727 31940 ts_tablet_manager.cc:1428] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:42.412847 31463 heartbeater.cc:503] Master 127.23.173.254:35793 was elected leader, sending a full tablet report...
I20260812 14:27:42.412825 31942 raft_consensus.cc:348] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.412987 31942 raft_consensus.cc:374] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:42.413038 31942 raft_consensus.cc:729] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192c0a30e9504498b475201d94400ed9, State: Initialized, Role: FOLLOWER
I20260812 14:27:42.413154 31942 consensus_queue.cc:260] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.413190 31950 tablet_bootstrap.cc:492] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Bootstrap starting.
I20260812 14:27:42.413460 31942 ts_tablet_manager.cc:1428] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:27:42.413462 31907 heartbeater.cc:503] Master 127.23.173.254:35793 was elected leader, sending a full tablet report...
I20260812 14:27:42.414062 31950 tablet_bootstrap.cc:654] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:42.414629 31950 log.cc:824] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:42.415287 31950 tablet_bootstrap.cc:492] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: No bootstrap required, opened a new log
I20260812 14:27:42.415356 31950 ts_tablet_manager.cc:1397] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:42.417001 31950 raft_consensus.cc:348] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.417136 31950 raft_consensus.cc:374] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:42.417205 31950 raft_consensus.cc:729] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6cd470e3cc489e82dc4fda542becad, State: Initialized, Role: FOLLOWER
I20260812 14:27:42.417328 31950 consensus_queue.cc:260] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.417595 31950 ts_tablet_manager.cc:1428] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:42.417621 31675 heartbeater.cc:503] Master 127.23.173.254:35793 was elected leader, sending a full tablet report...
W20260812 14:27:42.474478 31676 tablet.cc:2369] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:27:42.537510 31464 tablet.cc:2369] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:27:42.619789 31908 tablet.cc:2369] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:27:42.832453 31951 raft_consensus.cc:482] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:42.832600 31951 raft_consensus.cc:504] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.832983 31951 leader_election.cc:288] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f6cd470e3cc489e82dc4fda542becad (127.23.173.194:42265), d0292807e7b84a81bd0a84621a90e4fe (127.23.173.193:40489)
I20260812 14:27:42.837038 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.833243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:40489 (local address 127.23.173.195:47953)
0812 14:27:42.833327 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.833342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.833420 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.833591 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.833592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.833594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:42.833672 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.833674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.834611 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.834612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.835948 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.835952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.836741 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.836742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.836829 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.836834 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.836886 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.836933 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":15}
I20260812 14:27:42.837054 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.833314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:38283 (local address 127.23.173.194:42265)
0812 14:27:42.833426 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.833429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.833578 (+ 149us) server_negotiation.cc:407] Connection header received
0812 14:27:42.833613 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.833614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.833623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.833665 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:42.833805 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.834517 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.836072 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.836678 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.836752 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.836761 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.836929 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.836980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:27:42.837061 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.833396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:47953 (local address 127.23.173.193:40489)
0812 14:27:42.833461 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.833464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.833473 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:42.833503 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.833504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.833512 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.833572 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:42.833739 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.834513 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.836072 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.836642 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.836724 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.836735 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.836929 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.836980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:27:42.837044 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.833242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:42265 (local address 127.23.173.195:38283)
0812 14:27:42.833478 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.833487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:42.833567 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:42.833678 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:42.833679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:42.833680 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:42.833736 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.833737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.834611 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.834612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.835948 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:42.835952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.836750 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.836751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:42.836830 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.836834 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:42.836886 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:27:42.836933 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":99,"thread_start_us":133,"threads_started":1}
I20260812 14:27:42.837430 31393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bd614406d343a39be8a75543f8685b" candidate_uuid: "192c0a30e9504498b475201d94400ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0292807e7b84a81bd0a84621a90e4fe" is_pre_election: true
I20260812 14:27:42.837620 31393 raft_consensus.cc:2454] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 192c0a30e9504498b475201d94400ed9 in term 0.
I20260812 14:27:42.837690 31605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bd614406d343a39be8a75543f8685b" candidate_uuid: "192c0a30e9504498b475201d94400ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f6cd470e3cc489e82dc4fda542becad" is_pre_election: true
I20260812 14:27:42.837836 31605 raft_consensus.cc:2454] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 192c0a30e9504498b475201d94400ed9 in term 0.
I20260812 14:27:42.837874 31716 leader_election.cc:302] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192c0a30e9504498b475201d94400ed9, d0292807e7b84a81bd0a84621a90e4fe; no voters:
I20260812 14:27:42.838003 31951 raft_consensus.cc:2781] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:42.838086 31951 raft_consensus.cc:482] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:42.838124 31951 raft_consensus.cc:3037] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:42.838848 31951 raft_consensus.cc:504] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.838970 31951 leader_election.cc:288] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 4f6cd470e3cc489e82dc4fda542becad (127.23.173.194:42265), d0292807e7b84a81bd0a84621a90e4fe (127.23.173.193:40489)
I20260812 14:27:42.839151 31605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bd614406d343a39be8a75543f8685b" candidate_uuid: "192c0a30e9504498b475201d94400ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f6cd470e3cc489e82dc4fda542becad"
I20260812 14:27:42.839181 31393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bd614406d343a39be8a75543f8685b" candidate_uuid: "192c0a30e9504498b475201d94400ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0292807e7b84a81bd0a84621a90e4fe"
I20260812 14:27:42.839228 31605 raft_consensus.cc:3037] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:42.839264 31393 raft_consensus.cc:3037] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:42.840000 31605 raft_consensus.cc:2454] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192c0a30e9504498b475201d94400ed9 in term 1.
I20260812 14:27:42.840077 31393 raft_consensus.cc:2454] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192c0a30e9504498b475201d94400ed9 in term 1.
I20260812 14:27:42.840173 31720 leader_election.cc:302] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192c0a30e9504498b475201d94400ed9, 4f6cd470e3cc489e82dc4fda542becad; no voters:
I20260812 14:27:42.840265 31951 raft_consensus.cc:2781] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:42.840440 31951 raft_consensus.cc:686] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 192c0a30e9504498b475201d94400ed9, State: Running, Role: LEADER
I20260812 14:27:42.840570 31951 consensus_queue.cc:237] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } }
I20260812 14:27:42.841456 31138 catalog_manager.cc:5495] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 192c0a30e9504498b475201d94400ed9 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "192c0a30e9504498b475201d94400ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "192c0a30e9504498b475201d94400ed9" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:42.848011 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.845983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.23.173.195:36189)
0812 14:27:42.846028 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:42.846030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:42.846036 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:27:42.846064 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:42.846065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:42.846081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:42.846142 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:42.846252 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.846698 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.847377 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:42.847566 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:42.847616 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:42.847745 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:42.847841 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:42.847911 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:42.847922 (+ 11us) server_negotiation.cc:299] Negotiation successful
0812 14:27:42.847940 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:27:42.849714 31605 raft_consensus.cc:1261] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 1 FOLLOWER]: Refusing update from remote peer 192c0a30e9504498b475201d94400ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:42.849714 31393 raft_consensus.cc:1261] T 11bd614406d343a39be8a75543f8685b P d0292807e7b84a81bd0a84621a90e4fe [term 1 FOLLOWER]: Refusing update from remote peer 192c0a30e9504498b475201d94400ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:42.850033 31951 consensus_queue.cc:1035] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6cd470e3cc489e82dc4fda542becad" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:42.850172 31959 consensus_queue.cc:1035] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0292807e7b84a81bd0a84621a90e4fe" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:42.852550 31964 mvcc.cc:205] Tried to move back new op lower bound from 7317687758229774336 to 7317687758195433472. Current Snapshot: MvccSnapshot[applied={T|T < 7317687758229774336}]
I20260812 14:27:42.852726 31966 mvcc.cc:205] Tried to move back new op lower bound from 7317687758229774336 to 7317687758195433472. Current Snapshot: MvccSnapshot[applied={T|T < 7317687758229774336}]
I20260812 14:27:42.853066 31965 mvcc.cc:205] Tried to move back new op lower bound from 7317687758229774336 to 7317687758195433472. Current Snapshot: MvccSnapshot[applied={T|T < 7317687758229774336}]
I20260812 14:27:42.855201 31139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:42.855291 31139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:42.855999 31139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1358fb4f48984d28b0d15d318417b9ea: Sending DeleteTablet for 3 replicas of tablet 11bd614406d343a39be8a75543f8685b
I20260812 14:27:42.856331 31796 tablet_service.cc:1514] Processing DeleteTablet for tablet 11bd614406d343a39be8a75543f8685b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:42 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:60534
I20260812 14:27:42.856312 31570 tablet_service.cc:1514] Processing DeleteTablet for tablet 11bd614406d343a39be8a75543f8685b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:42 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:44470
I20260812 14:27:42.856345 31349 tablet_service.cc:1514] Processing DeleteTablet for tablet 11bd614406d343a39be8a75543f8685b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:42 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:33828
I20260812 14:27:42.856578 31980 tablet_replica.cc:330] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: stopping tablet replica
I20260812 14:27:42.856680 31980 raft_consensus.cc:2229] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:42.856793 31980 raft_consensus.cc:2258] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:42.856792 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 31252
I20260812 14:27:42.860575 31980 ts_tablet_manager.cc:1905] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:42.862535 31980 ts_tablet_manager.cc:1918] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:42.862612 31980 log.cc:1196] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/11bd614406d343a39be8a75543f8685b
I20260812 14:27:42.862847 31980 ts_tablet_manager.cc:1939] T 11bd614406d343a39be8a75543f8685b P 4f6cd470e3cc489e82dc4fda542becad: Deleting consensus metadata
I20260812 14:27:42.863099 31981 tablet_replica.cc:330] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: stopping tablet replica
I20260812 14:27:42.863199 31981 raft_consensus.cc:2229] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:42.863287 31119 catalog_manager.cc:4841] TS 4f6cd470e3cc489e82dc4fda542becad (127.23.173.194:42265): tablet 11bd614406d343a39be8a75543f8685b (table test-table [id=7bbd62869edf4cd7b46dfa8fb0fe4b9c]) successfully deleted
I20260812 14:27:42.863328 31981 raft_consensus.cc:2258] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:42.863811 31981 ts_tablet_manager.cc:1905] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:42.865384 31981 ts_tablet_manager.cc:1918] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:42.865463 31981 log.cc:1196] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/11bd614406d343a39be8a75543f8685b
I20260812 14:27:42.865694 31981 ts_tablet_manager.cc:1939] T 11bd614406d343a39be8a75543f8685b P 192c0a30e9504498b475201d94400ed9: Deleting consensus metadata
I20260812 14:27:42.866122 31118 catalog_manager.cc:4841] TS 192c0a30e9504498b475201d94400ed9 (127.23.173.195:36189): tablet 11bd614406d343a39be8a75543f8685b (table test-table [id=7bbd62869edf4cd7b46dfa8fb0fe4b9c]) successfully deleted
W20260812 14:27:42.868203 31117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:40489 (error 108)
I20260812 14:27:42.868724 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:42.868598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:40489 (local address 127.0.0.1:33834)
0812 14:27:42.868648 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:42.868689 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:27:42.868767 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 31468
W20260812 14:27:42.884014 31117 catalog_manager.cc:4584] TS d0292807e7b84a81bd0a84621a90e4fe (127.23.173.193:40489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11bd614406d343a39be8a75543f8685b: Network error: Client connection negotiation failed: client connection to 127.23.173.193:40489: connect: Connection refused (error 111)
I20260812 14:27:42.890087 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 31684
I20260812 14:27:42.905418 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 31097
2026-08-12T14:27:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2512 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T14:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:42Z Disabled control of system clock
I20260812 14:27:42.938680 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:34699
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:34699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:27:43.019403 31993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:43.019636 31993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:43.019656 31993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:43.020763 31993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:43.020797 31993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:43.020808 31993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:43.020819 31993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:43.020830 31993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:43.022185 31993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:34699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:34699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:43.022450 31993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:43.022810 31993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:43.028901 31999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.029063 32007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.029398 32003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.029672 31993 server_base.cc:1034] running on GCE node
I20260812 14:27:43.029908 31993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:43.030211 31993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:43.031361 31993 hybrid_clock.cc:648] HybridClock initialized: now 1786544863031338 us; error 41 us; skew 500 ppm
I20260812 14:27:43.032599 31993 webserver.cc:460] Webserver started at http://127.23.173.254:43003/ using document root <none> and password file <none>
I20260812 14:27:43.032835 31993 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:43.032883 31993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:43.033012 31993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:43.034083 31993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "229932044a9b4a11b2610b91fc232305"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "e932dda6f2092801eae2ec63ff036afe"
server_key_iv: "536fbe95beb5714dbe61ee66a15a6f23"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.034418 31993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "229932044a9b4a11b2610b91fc232305"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "e932dda6f2092801eae2ec63ff036afe"
server_key_iv: "536fbe95beb5714dbe61ee66a15a6f23"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.035799 31993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:43.036473 32013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.036631 31993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:43.036715 31993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "229932044a9b4a11b2610b91fc232305"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "e932dda6f2092801eae2ec63ff036afe"
server_key_iv: "536fbe95beb5714dbe61ee66a15a6f23"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.036784 31993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:43.040079 31993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:43.040331 31993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:43.040457 31993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:43.044186 31993 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:34699
I20260812 14:27:43.044226 32111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:34699 every 8 connection(s)
I20260812 14:27:43.044535 31993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:43.045194 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:43.048095 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305: Bootstrap starting.
I20260812 14:27:43.048619 32112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:43.048851 32112 log.cc:824] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:43.049451 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305: No bootstrap required, opened a new log
I20260812 14:27:43.050947 32112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229932044a9b4a11b2610b91fc232305" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34699 } }
I20260812 14:27:43.051102 32112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:43.051126 32112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229932044a9b4a11b2610b91fc232305, State: Initialized, Role: FOLLOWER
I20260812 14:27:43.051240 32112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229932044a9b4a11b2610b91fc232305" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34699 } }
I20260812 14:27:43.051313 32112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:43.051343 32112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:43.051373 32112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:43.051908 32112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229932044a9b4a11b2610b91fc232305" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34699 } }
I20260812 14:27:43.051994 32112 leader_election.cc:302] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229932044a9b4a11b2610b91fc232305; no voters:
I20260812 14:27:43.052130 32112 leader_election.cc:288] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:43.052189 32117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:43.052350 32117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [term 1 LEADER]: Becoming Leader. State: Replica: 229932044a9b4a11b2610b91fc232305, State: Running, Role: LEADER
I20260812 14:27:43.052353 32112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:43.052452 32117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229932044a9b4a11b2610b91fc232305" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34699 } }
I20260812 14:27:43.052826 32119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229932044a9b4a11b2610b91fc232305. Latest consensus state: current_term: 1 leader_uuid: "229932044a9b4a11b2610b91fc232305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229932044a9b4a11b2610b91fc232305" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34699 } } }
I20260812 14:27:43.052919 32119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:43.052940 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 31993
I20260812 14:27:43.053045 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:43.053124 32123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:43.053135 32118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229932044a9b4a11b2610b91fc232305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229932044a9b4a11b2610b91fc232305" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34699 } } }
I20260812 14:27:43.053246 24247 external_mini_cluster.cc:1428] Setting key c318f78cd823022bc0c8c649d52940d4
I20260812 14:27:43.053263 32118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:43.053665 32123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:43.055179 32123 catalog_manager.cc:1347] Generated new cluster ID: 2b96b042099741d29c176fe7bc891b2f
I20260812 14:27:43.055227 32123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:43.056383 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.053927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.23.173.254:34699)
0812 14:27:43.054120 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.054129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.054890 (+ 761us) server_negotiation.cc:407] Connection header received
0812 14:27:43.055025 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.055034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.055091 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.055240 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.055919 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.055922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.055923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.055927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.055975 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.055990 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.056062 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.056144 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.056200 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260812 14:27:43.075919 32123 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:43.076118 32123 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:43.079676 32123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305: Generated new TSK 0
I20260812 14:27:43.079898 32123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:43.081554 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34699
--builtin_ntp_servers=127.23.173.212:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:27:43.189486 32149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:43.189687 32149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:43.189707 32149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:43.190863 32149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:43.190905 32149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:43.190929 32149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:43.192342 32149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:43.192571 32149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:43.192858 32149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:43.197742 32163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.197750 32158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.197903 32165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.198491 32149 server_base.cc:1034] running on GCE node
I20260812 14:27:43.198679 32149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:43.198920 32149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:43.200078 32149 hybrid_clock.cc:648] HybridClock initialized: now 1786544863200068 us; error 28 us; skew 500 ppm
I20260812 14:27:43.201242 32149 webserver.cc:460] Webserver started at http://127.23.173.193:35829/ using document root <none> and password file <none>
I20260812 14:27:43.201490 32149 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:43.201545 32149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:43.201669 32149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:43.202704 32149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "2c2a6ae63495400e9f4596d3a74d4ad0"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "bdeff70a9b1d1ac7dd60696e7bc2181f"
server_key_iv: "c972521dbafdc76dbabad06c8ed6c841"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.203039 32149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "2c2a6ae63495400e9f4596d3a74d4ad0"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "bdeff70a9b1d1ac7dd60696e7bc2181f"
server_key_iv: "c972521dbafdc76dbabad06c8ed6c841"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.204321 32149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:43.204963 32176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.205143 32149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:43.205226 32149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "2c2a6ae63495400e9f4596d3a74d4ad0"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "bdeff70a9b1d1ac7dd60696e7bc2181f"
server_key_iv: "c972521dbafdc76dbabad06c8ed6c841"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.205281 32149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:43.208511 32149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:43.208724 32149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:43.208815 32149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:43.208963 32149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:43.209198 32149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:43.209227 32149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.209245 32149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:43.209261 32149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.214780 32149 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:39115
I20260812 14:27:43.214823 32359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:39115 every 8 connection(s)
I20260812 14:27:43.215123 32149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:43.217860 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.215829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:47969 (local address 127.23.173.254:34699)
0812 14:27:43.215891 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.215894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.216131 (+ 237us) server_negotiation.cc:407] Connection header received
0812 14:27:43.216558 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.216559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.216565 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.216619 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.217680 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.217682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.217682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.217686 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.217699 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.217703 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.217748 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.217797 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.217809 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:27:43.218061 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.215821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34699 (local address 127.23.173.193:47969)
0812 14:27:43.216013 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.216047 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.216483 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.216675 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.216678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.216686 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.217535 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.217535 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.217576 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.217604 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.217747 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.217749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.217751 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.217799 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.217843 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260812 14:27:43.218097 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 32149
I20260812 14:27:43.218189 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:43.218348 24247 external_mini_cluster.cc:1428] Setting key 97c5dd20b13730edf74a434451e83235
I20260812 14:27:43.218693 32360 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34699
I20260812 14:27:43.218772 32360 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:43.218911 32360 heartbeater.cc:511] Master 127.23.173.254:34699 requested a full tablet report, sending...
I20260812 14:27:43.219271 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34699
--builtin_ntp_servers=127.23.173.212:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 14:27:43.219457 32047 ts_manager.cc:194] Registered new tserver with Master: 2c2a6ae63495400e9f4596d3a74d4ad0 (127.23.173.193:39115)
W20260812 14:27:43.301136 32365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:43.301331 32365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:43.301352 32365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:43.302448 32365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:43.302480 32365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:43.302505 32365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:43.303843 32365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:43.304070 32365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:43.304358 32365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:43.309115 32372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.309139 32377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.309146 32370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.309320 32365 server_base.cc:1034] running on GCE node
I20260812 14:27:43.309504 32365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:43.309731 32365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:43.310870 32365 hybrid_clock.cc:648] HybridClock initialized: now 1786544863310855 us; error 28 us; skew 500 ppm
I20260812 14:27:43.312047 32365 webserver.cc:460] Webserver started at http://127.23.173.194:41007/ using document root <none> and password file <none>
I20260812 14:27:43.312270 32365 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:43.312319 32365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:43.312446 32365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:43.313463 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "fca3f4aa504377bbc3696c9debef7af3"
server_key_iv: "3e1c49a9234c6ff91f3b2393366a65a4"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.313824 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "fca3f4aa504377bbc3696c9debef7af3"
server_key_iv: "3e1c49a9234c6ff91f3b2393366a65a4"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.315037 32365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:43.315692 32386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.315874 32365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:43.315955 32365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "fca3f4aa504377bbc3696c9debef7af3"
server_key_iv: "3e1c49a9234c6ff91f3b2393366a65a4"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.316025 32365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:43.319221 32365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:43.319463 32365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:43.319589 32365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:43.319813 32365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:43.320103 32365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:43.320137 32365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.320163 32365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:43.320188 32365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.325639 32365 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:41355
I20260812 14:27:43.325676 32582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:41355 every 8 connection(s)
I20260812 14:27:43.325958 32365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:43.328773 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.326613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:38627 (local address 127.23.173.254:34699)
0812 14:27:43.326657 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.326660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.326854 (+ 194us) server_negotiation.cc:407] Connection header received
0812 14:27:43.327328 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.327329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.327336 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.327389 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.328525 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.328527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.328527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.328532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.328548 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.328553 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.328602 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.328682 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.328703 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:27:43.328938 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.326582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34699 (local address 127.23.173.194:38627)
0812 14:27:43.326741 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.326773 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.327229 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.327456 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.327460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.327469 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.328360 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.328360 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.328388 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.328416 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.328606 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.328612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.328615 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.328687 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.328747 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260812 14:27:43.329365 32583 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34699
I20260812 14:27:43.329438 32583 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:43.329572 32583 heartbeater.cc:511] Master 127.23.173.254:34699 requested a full tablet report, sending...
I20260812 14:27:43.329890 32047 ts_manager.cc:194] Registered new tserver with Master: ddaf58fc1c0b408d9bb5d1eaf32f59fc (127.23.173.194:41355)
I20260812 14:27:43.334697 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 32365
I20260812 14:27:43.334784 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:43.334940 24247 external_mini_cluster.cc:1428] Setting key d689de807a695d91e94346b7c1c550d9
I20260812 14:27:43.335898 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34699
--builtin_ntp_servers=127.23.173.212:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:27:43.416267 32586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:43.416472 32586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:43.416503 32586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:43.417640 32586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:43.417680 32586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:43.417713 32586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:43.419044 32586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:43.419276 32586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:43.419576 32586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:43.424490 32592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.424491 32598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.424548 32586 server_base.cc:1034] running on GCE node
W20260812 14:27:43.424667 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.424891 32586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:43.425132 32586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:43.426268 32586 hybrid_clock.cc:648] HybridClock initialized: now 1786544863426258 us; error 30 us; skew 500 ppm
I20260812 14:27:43.427503 32586 webserver.cc:460] Webserver started at http://127.23.173.195:36287/ using document root <none> and password file <none>
I20260812 14:27:43.427775 32586 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:43.427829 32586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:43.427963 32586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:43.428993 32586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "438ee756db6b4bcdb467c3a3b832bae2"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "cf1a30ca55f3ebdb0453d083038bcd24"
server_key_iv: "84815855786ba0c6a891e737bacbd891"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.429328 32586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "438ee756db6b4bcdb467c3a3b832bae2"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "cf1a30ca55f3ebdb0453d083038bcd24"
server_key_iv: "84815855786ba0c6a891e737bacbd891"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.430531 32586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:43.431231 32609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.431376 32586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:43.431456 32586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "438ee756db6b4bcdb467c3a3b832bae2"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "cf1a30ca55f3ebdb0453d083038bcd24"
server_key_iv: "84815855786ba0c6a891e737bacbd891"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.431525 32586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:43.434767 32586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:43.434996 32586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:43.435110 32586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:43.435298 32586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:43.435606 32586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:43.435644 32586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.435671 32586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:43.435696 32586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.441290 32586 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:35517
I20260812 14:27:43.441349 338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:35517 every 8 connection(s)
I20260812 14:27:43.441649 32586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:43.444525 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.442326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:57389 (local address 127.23.173.254:34699)
0812 14:27:43.442409 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.442411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.442623 (+ 212us) server_negotiation.cc:407] Connection header received
0812 14:27:43.443090 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.443091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.443097 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.443159 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.444307 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.444308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.444309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.444313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.444329 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.444333 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.444392 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.444444 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.444456 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 14:27:43.444684 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.442320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34699 (local address 127.23.173.195:57389)
0812 14:27:43.442506 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.442539 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.442979 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.443220 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.443223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.443232 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.444129 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.444130 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.444159 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.444188 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.444396 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.444399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.444401 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.444452 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.444501 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260812 14:27:43.445089 339 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34699
I20260812 14:27:43.445173 339 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:43.445290 339 heartbeater.cc:511] Master 127.23.173.254:34699 requested a full tablet report, sending...
I20260812 14:27:43.445613 32047 ts_manager.cc:194] Registered new tserver with Master: 438ee756db6b4bcdb467c3a3b832bae2 (127.23.173.195:35517)
I20260812 14:27:43.450965 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 32586
I20260812 14:27:43.451062 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:43.451248 24247 external_mini_cluster.cc:1428] Setting key e5301ae07fd9c1f12e79faa929a1e70e
I20260812 14:27:43.452178 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:27:43.454560 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.453513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.23.173.254:34699)
0812 14:27:43.453562 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.453564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.453696 (+ 132us) server_negotiation.cc:407] Connection header received
0812 14:27:43.453749 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.453749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.453754 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.453789 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.454381 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.454383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.454383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.454387 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.454403 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.454407 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.454457 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.454496 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.454508 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:27:43.456166 32047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:43.457638 32047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:43.461246 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.459630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41355 (local address 127.0.0.1:50674)
0812 14:27:43.459775 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.459814 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.459865 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.460203 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.460210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.460217 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.460816 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.460817 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.460823 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.460834 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.461086 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.461087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.461088 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.461142 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.461155 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260812 14:27:43.461243 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.459752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.23.173.194:41355)
0812 14:27:43.460018 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.460023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.460033 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:43.460073 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.460075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.460086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.460434 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.460920 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.460930 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.460930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.460935 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.460970 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.460995 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.461088 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.461150 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.461165 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260812 14:27:43.461295 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.459958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.23.173.195:35517)
0812 14:27:43.460129 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.460133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.460141 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:43.460164 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.460166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.460174 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.460430 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.460883 (+ 453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.460892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.460892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.460896 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.460932 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.460956 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.461018 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.461207 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.461222 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260812 14:27:43.461172 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.459868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35517 (local address 127.0.0.1:38240)
0812 14:27:43.459997 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.460005 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.460039 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.460266 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.460266 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.460268 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.460767 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.460768 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.460785 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.460811 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.461019 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.461025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.461028 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.461091 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.461104 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260812 14:27:43.461889 32464 tablet_service.cc:1467] Processing CreateTablet for tablet 2b248c006be74c37a0084c730e29ac25 (DEFAULT_TABLE table=test-table [id=eac3a57c40ce4a6c8f502f730a7da5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:43.462257 32464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b248c006be74c37a0084c730e29ac25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:43.462152 32695 tablet_service.cc:1467] Processing CreateTablet for tablet 2b248c006be74c37a0084c730e29ac25 (DEFAULT_TABLE table=test-table [id=eac3a57c40ce4a6c8f502f730a7da5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:43.462498 32695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b248c006be74c37a0084c730e29ac25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:43.464277 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.460058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39115 (local address 127.0.0.1:44374)
0812 14:27:43.460189 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.460197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.460236 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.461558 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.461559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.461561 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.463865 (+ 2304us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.463866 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.463872 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.463883 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.464131 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.464132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.464133 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.464194 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.464207 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260812 14:27:43.464309 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.460272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.23.173.193:39115)
0812 14:27:43.460902 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.460906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.460915 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:43.460939 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.460941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.460949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.461028 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.463972 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.463980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.463980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.463985 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.464030 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.464056 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.464122 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.464207 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.464222 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":86,"threads_started":1}
I20260812 14:27:43.465145 361 tablet_bootstrap.cc:492] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Bootstrap starting.
I20260812 14:27:43.465176 32257 tablet_service.cc:1467] Processing CreateTablet for tablet 2b248c006be74c37a0084c730e29ac25 (DEFAULT_TABLE table=test-table [id=eac3a57c40ce4a6c8f502f730a7da5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:43.465504 32257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b248c006be74c37a0084c730e29ac25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:43.465472 362 tablet_bootstrap.cc:492] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Bootstrap starting.
I20260812 14:27:43.465899 361 tablet_bootstrap.cc:654] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:43.466184 361 log.cc:824] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:43.466230 362 tablet_bootstrap.cc:654] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:43.466522 362 log.cc:824] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:43.466953 361 tablet_bootstrap.cc:492] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: No bootstrap required, opened a new log
I20260812 14:27:43.467031 361 ts_tablet_manager.cc:1397] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:43.467290 362 tablet_bootstrap.cc:492] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: No bootstrap required, opened a new log
I20260812 14:27:43.467363 362 ts_tablet_manager.cc:1397] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:43.468919 368 tablet_bootstrap.cc:492] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: Bootstrap starting.
I20260812 14:27:43.469197 361 raft_consensus.cc:348] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.469377 361 raft_consensus.cc:374] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:43.469412 361 raft_consensus.cc:729] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaf58fc1c0b408d9bb5d1eaf32f59fc, State: Initialized, Role: FOLLOWER
I20260812 14:27:43.469549 361 consensus_queue.cc:260] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.469620 368 tablet_bootstrap.cc:654] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:43.469739 362 raft_consensus.cc:348] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.469847 361 ts_tablet_manager.cc:1428] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:27:43.469867 362 raft_consensus.cc:374] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:43.469909 362 raft_consensus.cc:729] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438ee756db6b4bcdb467c3a3b832bae2, State: Initialized, Role: FOLLOWER
I20260812 14:27:43.469929 368 log.cc:824] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:43.469960 32583 heartbeater.cc:503] Master 127.23.173.254:34699 was elected leader, sending a full tablet report...
I20260812 14:27:43.470039 362 consensus_queue.cc:260] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.470292 339 heartbeater.cc:503] Master 127.23.173.254:34699 was elected leader, sending a full tablet report...
I20260812 14:27:43.470280 362 ts_tablet_manager.cc:1428] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:27:43.471315 368 tablet_bootstrap.cc:492] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: No bootstrap required, opened a new log
I20260812 14:27:43.471367 368 ts_tablet_manager.cc:1397] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 14:27:43.473112 368 raft_consensus.cc:348] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.473229 368 raft_consensus.cc:374] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:43.473248 368 raft_consensus.cc:729] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2a6ae63495400e9f4596d3a74d4ad0, State: Initialized, Role: FOLLOWER
I20260812 14:27:43.473322 368 consensus_queue.cc:260] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.473505 368 ts_tablet_manager.cc:1428] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:43.473543 32360 heartbeater.cc:503] Master 127.23.173.254:34699 was elected leader, sending a full tablet report...
I20260812 14:27:43.515291 374 raft_consensus.cc:482] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:43.515416 374 raft_consensus.cc:504] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.515794 374 leader_election.cc:288] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddaf58fc1c0b408d9bb5d1eaf32f59fc (127.23.173.194:41355), 2c2a6ae63495400e9f4596d3a74d4ad0 (127.23.173.193:39115)
I20260812 14:27:43.516971 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.516043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:56975 (local address 127.23.173.194:41355)
0812 14:27:43.516120 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.516122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.516132 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:43.516160 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.516161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.516166 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.516222 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.516744 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.516745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.516746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.516750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.516769 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.516774 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.516830 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.516891 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.516904 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 14:27:43.517004 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.515957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41355 (local address 127.23.173.195:56975)
0812 14:27:43.516010 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.516021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.516074 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.516235 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.516236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.516238 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.516642 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.516642 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.516650 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.516662 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.516838 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.516839 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.516840 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.516886 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.516900 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:27:43.517009 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.516170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:33513 (local address 127.23.173.193:39115)
0812 14:27:43.516223 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.516225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.516336 (+ 111us) server_negotiation.cc:407] Connection header received
0812 14:27:43.516363 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.516364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.516369 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.516422 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.516808 (+ 386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.516809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.516810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.516814 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.516830 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.516834 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.516884 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.516927 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.516941 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:43.517228 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.516077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39115 (local address 127.23.173.195:33513)
0812 14:27:43.516280 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.516289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.516329 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.516427 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.516428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.516430 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:43.516738 (+ 308us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:27:43.516738 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:43.516743 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:27:43.516752 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:43.516885 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:43.516886 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:43.516887 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:27:43.516928 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 14:27:43.516940 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":126,"threads_started":1}
I20260812 14:27:43.517318 32506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b248c006be74c37a0084c730e29ac25" candidate_uuid: "438ee756db6b4bcdb467c3a3b832bae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" is_pre_election: true
I20260812 14:27:43.517475 32506 raft_consensus.cc:2454] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 438ee756db6b4bcdb467c3a3b832bae2 in term 0.
I20260812 14:27:43.517549 32290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b248c006be74c37a0084c730e29ac25" candidate_uuid: "438ee756db6b4bcdb467c3a3b832bae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" is_pre_election: true
I20260812 14:27:43.517685 32290 raft_consensus.cc:2454] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 438ee756db6b4bcdb467c3a3b832bae2 in term 0.
I20260812 14:27:43.517747 32610 leader_election.cc:302] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 438ee756db6b4bcdb467c3a3b832bae2, ddaf58fc1c0b408d9bb5d1eaf32f59fc; no voters:
I20260812 14:27:43.517865 374 raft_consensus.cc:2781] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:43.517925 374 raft_consensus.cc:482] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:43.517946 374 raft_consensus.cc:3037] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:43.518730 374 raft_consensus.cc:504] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.518829 374 leader_election.cc:288] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [CANDIDATE]: Term 1 election: Requested vote from peers ddaf58fc1c0b408d9bb5d1eaf32f59fc (127.23.173.194:41355), 2c2a6ae63495400e9f4596d3a74d4ad0 (127.23.173.193:39115)
I20260812 14:27:43.519001 32290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b248c006be74c37a0084c730e29ac25" candidate_uuid: "438ee756db6b4bcdb467c3a3b832bae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0"
I20260812 14:27:43.519038 32506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b248c006be74c37a0084c730e29ac25" candidate_uuid: "438ee756db6b4bcdb467c3a3b832bae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc"
I20260812 14:27:43.519080 32290 raft_consensus.cc:3037] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:43.519119 32506 raft_consensus.cc:3037] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:43.519959 32290 raft_consensus.cc:2454] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 438ee756db6b4bcdb467c3a3b832bae2 in term 1.
I20260812 14:27:43.520067 32506 raft_consensus.cc:2454] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 438ee756db6b4bcdb467c3a3b832bae2 in term 1.
I20260812 14:27:43.520115 32610 leader_election.cc:302] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2a6ae63495400e9f4596d3a74d4ad0, 438ee756db6b4bcdb467c3a3b832bae2; no voters:
I20260812 14:27:43.520210 374 raft_consensus.cc:2781] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:43.520380 374 raft_consensus.cc:686] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 1 LEADER]: Becoming Leader. State: Replica: 438ee756db6b4bcdb467c3a3b832bae2, State: Running, Role: LEADER
I20260812 14:27:43.520488 374 consensus_queue.cc:237] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } }
I20260812 14:27:43.521281 32046 catalog_manager.cc:5495] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 438ee756db6b4bcdb467c3a3b832bae2 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "438ee756db6b4bcdb467c3a3b832bae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "438ee756db6b4bcdb467c3a3b832bae2" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:43.537504 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.536608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.23.173.195:35517)
0812 14:27:43.536664 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.536666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.536673 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:43.536724 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.536725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.536738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.536775 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.537285 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.537286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.537286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.537290 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.537306 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.537310 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.537355 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.537393 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.537405 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:27:43.539330 32506 raft_consensus.cc:1261] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 1 FOLLOWER]: Refusing update from remote peer 438ee756db6b4bcdb467c3a3b832bae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:43.539361 32290 raft_consensus.cc:1261] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 1 FOLLOWER]: Refusing update from remote peer 438ee756db6b4bcdb467c3a3b832bae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:43.539731 377 consensus_queue.cc:1035] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2a6ae63495400e9f4596d3a74d4ad0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:43.539865 374 consensus_queue.cc:1035] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaf58fc1c0b408d9bb5d1eaf32f59fc" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:43.542318 380 mvcc.cc:205] Tried to move back new op lower bound from 7317687761054535680 to 7317687760980635648. Current Snapshot: MvccSnapshot[applied={T|T < 7317687761054535680}]
I20260812 14:27:43.543411 382 mvcc.cc:205] Tried to move back new op lower bound from 7317687761054535680 to 7317687760980635648. Current Snapshot: MvccSnapshot[applied={T|T < 7317687761054535680}]
I20260812 14:27:43.543552 381 mvcc.cc:205] Tried to move back new op lower bound from 7317687761054535680 to 7317687760980635648. Current Snapshot: MvccSnapshot[applied={T|T < 7317687761054535680 or (T in {7317687761054535680})}]
I20260812 14:27:43.544811 32046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:43.544896 32046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:43.545604 32046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 229932044a9b4a11b2610b91fc232305: Sending DeleteTablet for 3 replicas of tablet 2b248c006be74c37a0084c730e29ac25
I20260812 14:27:43.545941 32464 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b248c006be74c37a0084c730e29ac25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:43 UTC) from {username='slave'} at 127.0.0.1:50674
I20260812 14:27:43.546017 32257 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b248c006be74c37a0084c730e29ac25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:43 UTC) from {username='slave'} at 127.0.0.1:44374
I20260812 14:27:43.546028 32695 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b248c006be74c37a0084c730e29ac25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:43 UTC) from {username='slave'} at 127.0.0.1:38240
I20260812 14:27:43.546171 400 tablet_replica.cc:330] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: stopping tablet replica
I20260812 14:27:43.546233 401 tablet_replica.cc:330] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: stopping tablet replica
I20260812 14:27:43.546262 400 raft_consensus.cc:2229] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:43.546322 401 raft_consensus.cc:2229] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:43.546363 400 raft_consensus.cc:2258] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:43.546411 401 raft_consensus.cc:2258] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:43.546617 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 32149
I20260812 14:27:43.546645 402 tablet_replica.cc:330] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: stopping tablet replica
I20260812 14:27:43.546802 402 raft_consensus.cc:2229] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:43.546850 401 ts_tablet_manager.cc:1905] T 2b248c006be74c37a0084c730e29ac25 P 2c2a6ae63495400e9f4596d3a74d4ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:43.546995 402 raft_consensus.cc:2258] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:43.547528 402 ts_tablet_manager.cc:1905] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:43.550750 402 ts_tablet_manager.cc:1918] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:43.550909 402 log.cc:1196] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/2b248c006be74c37a0084c730e29ac25
I20260812 14:27:43.551303 402 ts_tablet_manager.cc:1939] T 2b248c006be74c37a0084c730e29ac25 P 438ee756db6b4bcdb467c3a3b832bae2: Deleting consensus metadata
I20260812 14:27:43.552210 32015 catalog_manager.cc:4841] TS 438ee756db6b4bcdb467c3a3b832bae2 (127.23.173.195:35517): tablet 2b248c006be74c37a0084c730e29ac25 (table test-table [id=eac3a57c40ce4a6c8f502f730a7da5e7]) successfully deleted
I20260812 14:27:43.571378 400 ts_tablet_manager.cc:1905] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:43.573278 400 ts_tablet_manager.cc:1918] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:43.573378 400 log.cc:1196] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/2b248c006be74c37a0084c730e29ac25
I20260812 14:27:43.573683 400 ts_tablet_manager.cc:1939] T 2b248c006be74c37a0084c730e29ac25 P ddaf58fc1c0b408d9bb5d1eaf32f59fc: Deleting consensus metadata
I20260812 14:27:43.574144 32015 catalog_manager.cc:4841] TS ddaf58fc1c0b408d9bb5d1eaf32f59fc (127.23.173.194:41355): tablet 2b248c006be74c37a0084c730e29ac25 (table test-table [id=eac3a57c40ce4a6c8f502f730a7da5e7]) successfully deleted
W20260812 14:27:43.574319 32015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:39115 (error 108)
I20260812 14:27:43.574515 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 32365
I20260812 14:27:43.577087 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.574654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39115 (local address 127.0.0.1:44382)
0812 14:27:43.576981 (+ 2327us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.577045 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2296}
W20260812 14:27:43.586161 32015 catalog_manager.cc:4584] TS 2c2a6ae63495400e9f4596d3a74d4ad0 (127.23.173.193:39115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b248c006be74c37a0084c730e29ac25: Network error: Client connection negotiation failed: client connection to 127.23.173.193:39115: connect: Connection refused (error 111)
I20260812 14:27:43.590440 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 32586
I20260812 14:27:43.610867 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 31993
2026-08-12T14:27:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (709 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 14:27:43.640635 24247 mini_oidc.cc:152] Starting JWKS server
I20260812 14:27:43.641846 24247 webserver.cc:460] Webserver started at https://127.0.0.1:35255/ using document root <none> and password file <none>
I20260812 14:27:43.641927 24247 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:27:43.642071 24247 webserver.cc:460] Webserver started at http://127.0.0.1:37733/ using document root <none> and password file <none>
2026-08-12T14:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:43Z Disabled control of system clock
I20260812 14:27:43.650413 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:45181
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:45181
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35255/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/testchainca.cert with env {}
W20260812 14:27:43.730167 417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:43.730376 417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:43.730397 417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:43.731634 417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:43.731678 417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:43.731688 417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:43.731716 417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:43.731729 417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:27:43.731739 417 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:27:43.731750 417 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35255/jwks.json
I20260812 14:27:43.733300 417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:45181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35255/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:43.733498 417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:43.733661 417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:43.738200 424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.738210 425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.738291 428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.738402 417 server_base.cc:1034] running on GCE node
I20260812 14:27:43.738629 417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:43.738906 417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:43.740047 417 hybrid_clock.cc:648] HybridClock initialized: now 1786544863740041 us; error 40 us; skew 500 ppm
I20260812 14:27:43.741266 417 webserver.cc:460] Webserver started at http://127.23.173.254:40353/ using document root <none> and password file <none>
I20260812 14:27:43.741478 417 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:43.741524 417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:43.741652 417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:43.742662 417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "ef02fc4c11f34dd4a46aab8f3c276275"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "97c5559d7207938ce7cac0c0c7233388"
server_key_iv: "cc35d018b95184b9b5826173557066f6"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.742985 417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "ef02fc4c11f34dd4a46aab8f3c276275"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "97c5559d7207938ce7cac0c0c7233388"
server_key_iv: "cc35d018b95184b9b5826173557066f6"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.744192 417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:43.744894 440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.745061 417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:43.745147 417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "ef02fc4c11f34dd4a46aab8f3c276275"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "97c5559d7207938ce7cac0c0c7233388"
server_key_iv: "cc35d018b95184b9b5826173557066f6"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.745219 417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:43.752621 417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:43.752897 417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:43.753023 417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:43.756908 417 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:45181
I20260812 14:27:43.756940 521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:45181 every 8 connection(s)
I20260812 14:27:43.757257 417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:43.758170 522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:43.760545 522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275: Bootstrap starting.
I20260812 14:27:43.761148 522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:43.761420 522 log.cc:824] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:43.762087 522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275: No bootstrap required, opened a new log
I20260812 14:27:43.763659 522 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45181 } }
I20260812 14:27:43.763823 522 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:43.763854 522 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef02fc4c11f34dd4a46aab8f3c276275, State: Initialized, Role: FOLLOWER
I20260812 14:27:43.763998 522 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45181 } }
I20260812 14:27:43.764057 522 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:43.764086 522 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:43.764132 522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:43.764546 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 417
I20260812 14:27:43.764644 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:43.764683 522 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45181 } }
I20260812 14:27:43.764788 522 leader_election.cc:302] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef02fc4c11f34dd4a46aab8f3c276275; no voters:
I20260812 14:27:43.764820 24247 external_mini_cluster.cc:1428] Setting key bdef7fb7582db9a6cde0eaeaed0919a2
I20260812 14:27:43.764945 522 leader_election.cc:288] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:43.765025 529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:43.765152 529 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [term 1 LEADER]: Becoming Leader. State: Replica: ef02fc4c11f34dd4a46aab8f3c276275, State: Running, Role: LEADER
I20260812 14:27:43.765230 522 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:43.765241 529 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45181 } }
I20260812 14:27:43.765602 529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef02fc4c11f34dd4a46aab8f3c276275. Latest consensus state: current_term: 1 leader_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45181 } } }
I20260812 14:27:43.765676 529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:43.765702 532 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02fc4c11f34dd4a46aab8f3c276275" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45181 } } }
I20260812 14:27:43.765790 532 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:43.765969 540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:43.766469 540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:43.768213 540 catalog_manager.cc:1347] Generated new cluster ID: bdc8b439063045019eff4bb4a05979ea
I20260812 14:27:43.768267 540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:43.770558 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.765295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.23.173.254:45181)
0812 14:27:43.765652 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.765662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.765781 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:27:43.765950 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.765957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.766079 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.766228 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:43.766511 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:43.767830 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:43.768713 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:43.769142 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:43.769251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:43.769911 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:43.769915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:43.769916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:43.770093 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:43.770127 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:43.770137 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:43.770194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:43.770277 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:27:43.770355 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260812 14:27:43.771416 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:43.791975 540 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:43.792680 540 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:43.799203 540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef02fc4c11f34dd4a46aab8f3c276275: Generated new TSK 0
I20260812 14:27:43.799381 540 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 14:27:43.823997 577 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 14:27:43.824152 577 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 14:27:43.824304 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:43.808804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.23.173.254:45181)
0812 14:27:43.808848 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:43.808851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:43.808982 (+ 131us) server_negotiation.cc:407] Connection header received
0812 14:27:43.809074 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:43.809075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:43.809091 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:43.809156 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:27:43.809306 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:43.809802 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:43.810908 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:43.811108 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:43.811166 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:43.811373 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:27:43.824065 (+ 12692us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 14:27:43.824238 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:27:43.824360 535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 14:27:43.824862 576 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:43.808718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45181 (local address 127.0.0.1:41650)
0812 14:27:43.808887 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:43.808899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:43.808977 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:43.809174 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:43.809175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:43.809177 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:27:43.809246 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:43.809248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:43.809872 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:43.809873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:43.810814 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:43.810816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:43.811185 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:43.811186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:43.811250 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:43.811274 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:27:43.824315 (+ 13041us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 14:27:43.824449 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.254:45181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.173.254:45181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 14:27:43.825336 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 417
2026-08-12T14:27:43Z chronyd exiting
I20260812 14:27:43.870452 24247 test_util.cc:175] -----------------------------------------------
I20260812 14:27:43.870544 24247 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544840008271-24247-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 14:27:43.871304 24247 mini_oidc.cc:152] Starting JWKS server
I20260812 14:27:43.872062 24247 webserver.cc:460] Webserver started at https://127.0.0.1:42225/ using document root <none> and password file <none>
I20260812 14:27:43.872105 24247 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:27:43.872248 24247 webserver.cc:460] Webserver started at http://127.0.0.1:45857/ using document root <none> and password file <none>
2026-08-12T14:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:43Z Disabled control of system clock
I20260812 14:27:43.880630 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35327
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42225/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/testchainca.cert with env {}
W20260812 14:27:43.962361 587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:43.962586 587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:43.962607 587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:43.963730 587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:43.963766 587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:43.963778 587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:43.963789 587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:43.963799 587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:27:43.963814 587 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:27:43.963827 587 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42225/jwks.json
I20260812 14:27:43.965288 587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42225/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:43.965500 587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:43.965672 587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:43.970284 593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.970302 594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:43.970302 598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:43.970806 587 server_base.cc:1034] running on GCE node
I20260812 14:27:43.971032 587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:43.971271 587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:43.972402 587 hybrid_clock.cc:648] HybridClock initialized: now 1786544863972387 us; error 28 us; skew 500 ppm
I20260812 14:27:43.973582 587 webserver.cc:460] Webserver started at http://127.23.173.254:36297/ using document root <none> and password file <none>
I20260812 14:27:43.973802 587 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:43.973846 587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:43.973964 587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:43.974987 587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "864a111819494a1f88348a0a2a91bb7c"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "a688d1645c9542e8bcb2422d06159579"
server_key_iv: "22424367c5a6b08d11e7d015431bf2e3"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.975308 587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "864a111819494a1f88348a0a2a91bb7c"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "a688d1645c9542e8bcb2422d06159579"
server_key_iv: "22424367c5a6b08d11e7d015431bf2e3"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.977105 587 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 14:27:43.977857 608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:43.978009 587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:43.978101 587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "864a111819494a1f88348a0a2a91bb7c"
format_stamp: "Formatted at 2026-08-12 14:27:43 on dist-test-slave-1l3l"
server_key: "a688d1645c9542e8bcb2422d06159579"
server_key_iv: "22424367c5a6b08d11e7d015431bf2e3"
server_key_version: "encryptionkey@0"
I20260812 14:27:43.978168 587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:43.997274 587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:43.997617 587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:43.997756 587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:44.001698 587 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:35327
I20260812 14:27:44.001732 700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:35327 every 8 connection(s)
I20260812 14:27:44.002060 587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:44.002699 701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:44.004791 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 587
I20260812 14:27:44.004904 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:44.005095 24247 external_mini_cluster.cc:1428] Setting key 8ca2fb4e76bf68c2969868072c3fbf53
I20260812 14:27:44.005800 701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c: Bootstrap starting.
I20260812 14:27:44.006400 701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:44.006670 701 log.cc:824] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:44.007324 701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c: No bootstrap required, opened a new log
I20260812 14:27:44.009110 701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a111819494a1f88348a0a2a91bb7c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35327 } }
I20260812 14:27:44.009258 701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:44.009302 701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864a111819494a1f88348a0a2a91bb7c, State: Initialized, Role: FOLLOWER
I20260812 14:27:44.009424 701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a111819494a1f88348a0a2a91bb7c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35327 } }
I20260812 14:27:44.009503 701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:44.009547 701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:44.009598 701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:44.010119 701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a111819494a1f88348a0a2a91bb7c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35327 } }
I20260812 14:27:44.010233 701 leader_election.cc:302] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 864a111819494a1f88348a0a2a91bb7c; no voters:
I20260812 14:27:44.010373 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:44.005681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.23.173.254:35327)
0812 14:27:44.005857 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:44.005866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:44.005900 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:27:44.006330 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:44.006336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:44.006405 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:44.006544 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:44.006739 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.007665 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.008473 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.008874 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.008992 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.009707 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:44.009711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:44.009711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:44.009884 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:44.009917 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:44.009927 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:44.009981 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:44.010066 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:27:44.010144 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260812 14:27:44.010383 701 leader_election.cc:288] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:44.010433 709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:44.010535 709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [term 1 LEADER]: Becoming Leader. State: Replica: 864a111819494a1f88348a0a2a91bb7c, State: Running, Role: LEADER
I20260812 14:27:44.010619 709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a111819494a1f88348a0a2a91bb7c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35327 } }
I20260812 14:27:44.010660 701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:44.010914 709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 864a111819494a1f88348a0a2a91bb7c. Latest consensus state: current_term: 1 leader_uuid: "864a111819494a1f88348a0a2a91bb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a111819494a1f88348a0a2a91bb7c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35327 } } }
I20260812 14:27:44.010977 709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:44.011157 710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "864a111819494a1f88348a0a2a91bb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a111819494a1f88348a0a2a91bb7c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35327 } } }
I20260812 14:27:44.011234 710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:44.011319 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:44.011338 714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:44.011861 714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:44.013532 714 catalog_manager.cc:1347] Generated new cluster ID: 15f5b368785541efb456cd7ed48c24e2
I20260812 14:27:44.013592 714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:44.027987 714 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:44.028494 714 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:44.037431 714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 864a111819494a1f88348a0a2a91bb7c: Generated new TSK 0
I20260812 14:27:44.037633 714 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 14:27:44.059661 748 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 14:27:44.059800 748 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 14:27:44.060199 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:44.045862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.23.173.254:35327)
0812 14:27:44.045917 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:44.045919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:44.045978 (+ 59us) server_negotiation.cc:407] Connection header received
0812 14:27:44.046036 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:44.046037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:44.046049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:44.046137 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:27:44.046237 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.047013 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.047775 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.048119 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.048185 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.048321 (+ 136us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:27:44.059934 (+ 11613us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 14:27:44.060121 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260812 14:27:44.060272 706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 14:27:44.060397 746 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:44.045771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35327 (local address 127.0.0.1:52252)
0812 14:27:44.045913 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:44.045923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:44.045978 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:44.046133 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:44.046134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:44.046135 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:27:44.046188 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:44.046190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.047081 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.047082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:44.047669 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:44.047671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.048184 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.048185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:44.048225 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.048230 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:27:44.060105 (+ 11875us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 14:27:44.060225 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.254:35327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260812 14:27:44.060694 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 587
2026-08-12T14:27:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 14:27:44.103039 24247 mini_oidc.cc:152] Starting JWKS server
I20260812 14:27:44.103770 24247 webserver.cc:460] Webserver started at https://127.0.0.1:40111/ using document root <none> and password file <none>
I20260812 14:27:44.103820 24247 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:27:44.103965 24247 webserver.cc:460] Webserver started at http://127.0.0.1:40523/ using document root <none> and password file <none>
2026-08-12T14:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:44Z Disabled control of system clock
I20260812 14:27:44.113137 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43063
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:43063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 14:27:44.196192 757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:44.196416 757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:44.196439 757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:44.197579 757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:44.197615 757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:44.197628 757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:44.197638 757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:44.197649 757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:27:44.197659 757 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:27:44.197669 757 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40111/jwks.json
I20260812 14:27:44.198999 757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:43063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:44.199211 757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:44.199468 757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:44.204448 764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:44.204483 757 server_base.cc:1034] running on GCE node
W20260812 14:27:44.204586 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:44.204689 763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:44.204953 757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:44.205215 757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:44.206368 757 hybrid_clock.cc:648] HybridClock initialized: now 1786544864206356 us; error 29 us; skew 500 ppm
I20260812 14:27:44.207645 757 webserver.cc:460] Webserver started at http://127.23.173.254:33247/ using document root <none> and password file <none>
I20260812 14:27:44.207909 757 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:44.207966 757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:44.208099 757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:44.209192 757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "e05adac65a90451d82ceea326611e0c6"
format_stamp: "Formatted at 2026-08-12 14:27:44 on dist-test-slave-1l3l"
server_key: "8f04ef6c9938ff68741733fd2d1828d4"
server_key_iv: "a66ad9fc76e0debfea4203639c0b0498"
server_key_version: "encryptionkey@0"
I20260812 14:27:44.209547 757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "e05adac65a90451d82ceea326611e0c6"
format_stamp: "Formatted at 2026-08-12 14:27:44 on dist-test-slave-1l3l"
server_key: "8f04ef6c9938ff68741733fd2d1828d4"
server_key_iv: "a66ad9fc76e0debfea4203639c0b0498"
server_key_version: "encryptionkey@0"
I20260812 14:27:44.210839 757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:44.211603 775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:44.211808 757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:44.211905 757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "e05adac65a90451d82ceea326611e0c6"
format_stamp: "Formatted at 2026-08-12 14:27:44 on dist-test-slave-1l3l"
server_key: "8f04ef6c9938ff68741733fd2d1828d4"
server_key_iv: "a66ad9fc76e0debfea4203639c0b0498"
server_key_version: "encryptionkey@0"
I20260812 14:27:44.211968 757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:44.219165 757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:44.219473 757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:44.219611 757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:44.223508 757 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:43063
I20260812 14:27:44.223546 864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:43063 every 8 connection(s)
I20260812 14:27:44.223920 757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:44.224669 865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:44.227322 865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6: Bootstrap starting.
I20260812 14:27:44.227923 865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:44.228152 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 757
I20260812 14:27:44.228216 865 log.cc:824] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:44.228256 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:44.228416 24247 external_mini_cluster.cc:1428] Setting key a52ec546b312d5425e3d19d7073202fe
I20260812 14:27:44.229049 865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6: No bootstrap required, opened a new log
I20260812 14:27:44.231940 865 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05adac65a90451d82ceea326611e0c6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43063 } }
I20260812 14:27:44.232097 865 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:44.232120 865 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05adac65a90451d82ceea326611e0c6, State: Initialized, Role: FOLLOWER
I20260812 14:27:44.232239 865 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05adac65a90451d82ceea326611e0c6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43063 } }
I20260812 14:27:44.232313 865 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:44.232337 865 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:44.232383 865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:44.233191 865 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05adac65a90451d82ceea326611e0c6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43063 } }
I20260812 14:27:44.233305 865 leader_election.cc:302] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05adac65a90451d82ceea326611e0c6; no voters:
I20260812 14:27:44.233479 865 leader_election.cc:288] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:44.233525 874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:44.233665 874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [term 1 LEADER]: Becoming Leader. State: Replica: e05adac65a90451d82ceea326611e0c6, State: Running, Role: LEADER
I20260812 14:27:44.233773 865 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:44.233763 874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05adac65a90451d82ceea326611e0c6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43063 } }
I20260812 14:27:44.234210 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05adac65a90451d82ceea326611e0c6. Latest consensus state: current_term: 1 leader_uuid: "e05adac65a90451d82ceea326611e0c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05adac65a90451d82ceea326611e0c6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43063 } } }
I20260812 14:27:44.234292 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:44.234519 889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:44.235008 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:44.228931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.23.173.254:43063)
0812 14:27:44.229152 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:44.229162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:44.229201 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:27:44.229703 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:44.229714 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:44.229796 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:44.229950 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:44.230119 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.231458 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.232449 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.233057 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.233189 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.233947 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:44.233952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:44.233952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:44.234169 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:44.234215 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:44.234232 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:44.234294 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:44.234716 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 14:27:44.234809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260812 14:27:44.235278 889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:44.235652 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05adac65a90451d82ceea326611e0c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05adac65a90451d82ceea326611e0c6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43063 } } }
I20260812 14:27:44.235764 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:44.235999 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:44.237922 889 catalog_manager.cc:1347] Generated new cluster ID: f0765ce43586458482f404f775e1f198
I20260812 14:27:44.237978 889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:44.248558 889 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:44.249097 889 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:44.259343 889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05adac65a90451d82ceea326611e0c6: Generated new TSK 0
I20260812 14:27:44.259564 889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:44.270953 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:44.267050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.23.173.254:43063)
0812 14:27:44.267095 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:44.267097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:44.267244 (+ 147us) server_negotiation.cc:407] Connection header received
0812 14:27:44.267323 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:44.267324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:44.267334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:44.267396 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:27:44.267547 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.268127 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.269301 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.269506 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.269560 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.269794 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:27:44.270726 (+ 932us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 14:27:44.270865 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:27:44.271026 873 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 14:27:44.271227 922 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:44.266993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:43063 (local address 127.0.0.1:40720)
0812 14:27:44.267166 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:44.267179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:44.267254 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:44.267410 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:44.267412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:44.267414 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:27:44.267486 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:44.267488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.268201 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.268202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:44.269192 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:44.269195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.269579 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.269580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:44.269649 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.269664 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:27:44.270828 (+ 1164us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 14:27:44.270948 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.254:43063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260812 14:27:44.271631 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 757
E20260812 14:27:44.289633 924 webserver.cc:533] Webserver: sslize failed:
E20260812 14:27:44.289716 924 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T14:27:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T14:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:44Z Disabled control of system clock
I20260812 14:27:44.328939 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:44397
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:44397
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/testchainca.cert with env {}
W20260812 14:27:44.408263 934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:44.408475 934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:44.408504 934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:44.409646 934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:44.409687 934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:44.409704 934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:44.409719 934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:44.409739 934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:27:44.409757 934 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:27:44.409778 934 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 14:27:44.411240 934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:44397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:44397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:44.411476 934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:44.411674 934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:44.416442 948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:44.416453 945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:44.416494 942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:44.416563 934 server_base.cc:1034] running on GCE node
I20260812 14:27:44.416982 934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:44.417281 934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:44.418452 934 hybrid_clock.cc:648] HybridClock initialized: now 1786544864418434 us; error 32 us; skew 500 ppm
I20260812 14:27:44.419817 934 webserver.cc:460] Webserver started at http://127.23.173.254:43183/ using document root <none> and password file <none>
I20260812 14:27:44.420056 934 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:44.420105 934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:44.420231 934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:44.421336 934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "f81f152dab7345e09f72d312dd73c8b9"
format_stamp: "Formatted at 2026-08-12 14:27:44 on dist-test-slave-1l3l"
server_key: "a244f735b4e89deeb2e366fe5c72b8df"
server_key_iv: "08fff2d408ab9742a81d2324aa8bbb43"
server_key_version: "encryptionkey@0"
I20260812 14:27:44.421676 934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "f81f152dab7345e09f72d312dd73c8b9"
format_stamp: "Formatted at 2026-08-12 14:27:44 on dist-test-slave-1l3l"
server_key: "a244f735b4e89deeb2e366fe5c72b8df"
server_key_iv: "08fff2d408ab9742a81d2324aa8bbb43"
server_key_version: "encryptionkey@0"
I20260812 14:27:44.422950 934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:44.423686 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:44.423856 934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:44.423942 934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "f81f152dab7345e09f72d312dd73c8b9"
format_stamp: "Formatted at 2026-08-12 14:27:44 on dist-test-slave-1l3l"
server_key: "a244f735b4e89deeb2e366fe5c72b8df"
server_key_iv: "08fff2d408ab9742a81d2324aa8bbb43"
server_key_version: "encryptionkey@0"
I20260812 14:27:44.424011 934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:44.453251 934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:44.453611 934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:44.453770 934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:44.457721 934 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:44397
I20260812 14:27:44.457794 1050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:44397 every 8 connection(s)
I20260812 14:27:44.458124 934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:44.459111 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:44.461477 1054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9: Bootstrap starting.
I20260812 14:27:44.462113 1054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:44.462375 1054 log.cc:824] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:44.463078 1054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9: No bootstrap required, opened a new log
I20260812 14:27:44.463625 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 934
I20260812 14:27:44.463766 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:44.463963 24247 external_mini_cluster.cc:1428] Setting key 886edd1f9ec2b7c498c94cd4765892f5
I20260812 14:27:44.465103 1054 raft_consensus.cc:348] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81f152dab7345e09f72d312dd73c8b9" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44397 } }
I20260812 14:27:44.465245 1054 raft_consensus.cc:374] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:44.465279 1054 raft_consensus.cc:729] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81f152dab7345e09f72d312dd73c8b9, State: Initialized, Role: FOLLOWER
I20260812 14:27:44.465423 1054 consensus_queue.cc:260] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81f152dab7345e09f72d312dd73c8b9" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44397 } }
I20260812 14:27:44.465492 1054 raft_consensus.cc:388] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:44.465520 1054 raft_consensus.cc:482] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:44.465569 1054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:44.466171 1054 raft_consensus.cc:504] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81f152dab7345e09f72d312dd73c8b9" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44397 } }
I20260812 14:27:44.466289 1054 leader_election.cc:302] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f81f152dab7345e09f72d312dd73c8b9; no voters:
I20260812 14:27:44.466451 1054 leader_election.cc:288] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:44.466483 1062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:44.466611 1062 raft_consensus.cc:686] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [term 1 LEADER]: Becoming Leader. State: Replica: f81f152dab7345e09f72d312dd73c8b9, State: Running, Role: LEADER
I20260812 14:27:44.466696 1062 consensus_queue.cc:237] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81f152dab7345e09f72d312dd73c8b9" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44397 } }
I20260812 14:27:44.466729 1054 sys_catalog.cc:564] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:44.467054 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f81f152dab7345e09f72d312dd73c8b9. Latest consensus state: current_term: 1 leader_uuid: "f81f152dab7345e09f72d312dd73c8b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81f152dab7345e09f72d312dd73c8b9" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44397 } } }
I20260812 14:27:44.467144 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:44.467069 1063 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f81f152dab7345e09f72d312dd73c8b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81f152dab7345e09f72d312dd73c8b9" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44397 } } }
I20260812 14:27:44.467290 1063 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9 [sys.catalog]: This master's current role is: LEADER
W20260812 14:27:44.468621 1089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:44.468722 1089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:44.468822 1072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:44.468990 1072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:44.470059 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:44.464477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.23.173.254:44397)
0812 14:27:44.464701 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:44.464709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:44.464733 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:27:44.465071 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:44.465077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:44.465132 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:44.465251 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:44.465458 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.466906 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.468006 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.468513 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.468635 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.469308 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:44.469313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:44.469313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:44.469540 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:44.469586 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:44.469601 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:44.469677 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:44.469765 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:27:44.469846 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260812 14:27:44.470501 1072 catalog_manager.cc:1347] Generated new cluster ID: 52e14812bec148d68292424f610d8183
I20260812 14:27:44.470557 1072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:44.471149 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:44.476953 1072 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:44.477440 1072 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:44.481003 1072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f81f152dab7345e09f72d312dd73c8b9: Generated new TSK 0
I20260812 14:27:44.481163 1072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:44.548219 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:44.488566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.23.173.254:44397)
0812 14:27:44.488610 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:44.488612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:44.488648 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:27:44.488708 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:44.488708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:44.488721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:44.488797 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:27:44.488905 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.489657 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.490613 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.490927 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.490991 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.491106 (+ 115us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:27:44.547919 (+ 56813us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 14:27:44.548136 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56326: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 14:27:44.548286 1105 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:44.488439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44397 (local address 127.0.0.1:56326)
0812 14:27:44.488592 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:44.488602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:44.488658 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:44.488793 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:44.488794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:44.488795 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:27:44.488847 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:44.488849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.489962 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.489963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:44.490543 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:44.490545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:44.490992 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:44.490992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:44.491028 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:44.491033 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:27:44.548075 (+ 57042us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 14:27:44.548201 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.254:44397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260812 14:27:44.548312 1060 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56326: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260812 14:27:44.548731 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 934
2026-08-12T14:27:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (248 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:44 dist-test-slave-1l3l krb5kdc[1122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:44 dist-test-slave-1l3l krb5kdc[1122](info): set up 2 sockets
Aug 12 14:27:44 dist-test-slave-1l3l krb5kdc[1122](info): commencing operation
krb5kdc: starting...
W20260812 14:27:45.911115 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:45 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:27:46.029937 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46015
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:46.113196 1143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:46.113420 1143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:46.113451 1143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:46.114621 1143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:46.114670 1143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:46.114686 1143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:46.114703 1143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:46.114718 1143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:46.116288 1143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:46.116572 1143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:46.116798 1143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:46.121739 1153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:46.121711 1151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.121750 1143 server_base.cc:1034] running on GCE node
W20260812 14:27:46.121717 1155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.122208 1143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:46.122510 1143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:46.123664 1143 hybrid_clock.cc:648] HybridClock initialized: now 1786544866123632 us; error 46 us; skew 500 ppm
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.126168 1143 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:27:46.126634 1143 webserver.cc:460] Webserver started at http://127.23.173.254:39261/ using document root <none> and password file <none>
I20260812 14:27:46.126829 1143 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:46.126869 1143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:46.126963 1143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:46.128044 1143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "28c35b3898ad4e4ebccd12909d635408"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "b319005ad129ff8d8e22f35ae403dc3c"
server_key_iv: "4e81feaca4e549d68b19ed068cb2df7e"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.128345 1143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "28c35b3898ad4e4ebccd12909d635408"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "b319005ad129ff8d8e22f35ae403dc3c"
server_key_iv: "4e81feaca4e549d68b19ed068cb2df7e"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.129575 1143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:46.130221 1164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.130419 1143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:46.130517 1143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "28c35b3898ad4e4ebccd12909d635408"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "b319005ad129ff8d8e22f35ae403dc3c"
server_key_iv: "4e81feaca4e549d68b19ed068cb2df7e"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.130584 1143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:46.162317 1143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:46.164726 1143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:46.164885 1143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:46.168955 1143 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46015
I20260812 14:27:46.169011 1243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46015 every 8 connection(s)
I20260812 14:27:46.169348 1143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:46.170233 1244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:46.172645 1244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408: Bootstrap starting.
I20260812 14:27:46.173243 1244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:46.173521 1244 log.cc:824] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:46.174276 1244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408: No bootstrap required, opened a new log
I20260812 14:27:46.176085 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 1143
I20260812 14:27:46.176020 1244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c35b3898ad4e4ebccd12909d635408" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46015 } }
I20260812 14:27:46.176178 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:46.176168 1244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:46.176208 1244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c35b3898ad4e4ebccd12909d635408, State: Initialized, Role: FOLLOWER
I20260812 14:27:46.176365 24247 external_mini_cluster.cc:1428] Setting key 99332a70fb03d5a7a408d970ce29f616
I20260812 14:27:46.176358 1244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c35b3898ad4e4ebccd12909d635408" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46015 } }
I20260812 14:27:46.176445 1244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:46.176482 1244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:46.176532 1244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:46.177250 1244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c35b3898ad4e4ebccd12909d635408" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46015 } }
I20260812 14:27:46.177356 1244 leader_election.cc:302] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c35b3898ad4e4ebccd12909d635408; no voters:
I20260812 14:27:46.177533 1244 leader_election.cc:288] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:46.177577 1259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:46.177727 1259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [term 1 LEADER]: Becoming Leader. State: Replica: 28c35b3898ad4e4ebccd12909d635408, State: Running, Role: LEADER
I20260812 14:27:46.177812 1244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:46.177898 1259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c35b3898ad4e4ebccd12909d635408" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46015 } }
I20260812 14:27:46.178278 1260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c35b3898ad4e4ebccd12909d635408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c35b3898ad4e4ebccd12909d635408" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46015 } } }
I20260812 14:27:46.178360 1260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:46.178586 1261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c35b3898ad4e4ebccd12909d635408. Latest consensus state: current_term: 1 leader_uuid: "28c35b3898ad4e4ebccd12909d635408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c35b3898ad4e4ebccd12909d635408" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46015 } } }
I20260812 14:27:46.178925 1261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408 [sys.catalog]: This master's current role is: LEADER
W20260812 14:27:46.179804 1283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:46.179859 1283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:46.179900 1271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:46.180050 1271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:46.184166 1271 catalog_manager.cc:1347] Generated new cluster ID: 84a2b687a7464f369758ad9d490688d6
I20260812 14:27:46.184228 1271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:46.186736 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.176904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.23.173.254:46015)
0812 14:27:46.177370 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.177380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.177415 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:27:46.177537 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.177544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.177598 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.178549 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.178655 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.179860 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.180645 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.181157 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.181253 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.183541 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.183545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.183545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.183559 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.185207 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.185540 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.185543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.185544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.185648 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.185827 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.185828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.185828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.186084 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.186231 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.186334 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.186569 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":359,"threads_started":1}
I20260812 14:27:46.192991 1271 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:46.193703 1271 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:46.199848 1271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c35b3898ad4e4ebccd12909d635408: Generated new TSK 0
I20260812 14:27:46.200137 1271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:27:46.231338 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46015
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:46.316221 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:46.316452 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:46.316488 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:46.317648 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:46.317690 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:46.317725 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:46.319221 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:46.319460 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:46.319685 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:46.324651 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:46.324702 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.324795 1291 server_base.cc:1034] running on GCE node
W20260812 14:27:46.324651 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.325115 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:46.325378 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:46.326519 1291 hybrid_clock.cc:648] HybridClock initialized: now 1786544866326504 us; error 34 us; skew 500 ppm
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.329039 1291 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:27:46.329447 1291 webserver.cc:460] Webserver started at http://127.23.173.193:34489/ using document root <none> and password file <none>
I20260812 14:27:46.329676 1291 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:46.329723 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:46.329844 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:46.330873 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "a0ccb0837b6e4ce58b575439459ba956"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "2325ecf0a67e4519b62f6b9f8ea28b17"
server_key_iv: "7066b54cdcaf82620baa7229cd6a21b8"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.331205 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "a0ccb0837b6e4ce58b575439459ba956"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "2325ecf0a67e4519b62f6b9f8ea28b17"
server_key_iv: "7066b54cdcaf82620baa7229cd6a21b8"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.332526 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:46.333267 1315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.333451 1291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:46.333526 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "a0ccb0837b6e4ce58b575439459ba956"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "2325ecf0a67e4519b62f6b9f8ea28b17"
server_key_iv: "7066b54cdcaf82620baa7229cd6a21b8"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.333606 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:46.341589 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:46.343302 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:46.343415 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:46.343644 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:46.343983 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:46.344019 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.344039 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:46.344062 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.349704 1291 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:33161
I20260812 14:27:46.349763 1520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:33161 every 8 connection(s)
I20260812 14:27:46.350062 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:46.357934 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 1291
I20260812 14:27:46.358021 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:46.358201 24247 external_mini_cluster.cc:1428] Setting key 090fc6da8c546f339c0541b5a488a13d
I20260812 14:27:46.359205 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.350786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:40999 (local address 127.23.173.254:46015)
0812 14:27:46.350839 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.350841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.351021 (+ 180us) server_negotiation.cc:407] Connection header received
0812 14:27:46.352074 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.352076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.352095 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.352150 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.352791 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.353300 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.354273 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.354483 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.354541 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.356599 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.356601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.356601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.356610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.357787 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.358245 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.358246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.358247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.358313 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.358548 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.358549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.358549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.358675 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.358744 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.359050 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.359140 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:27:46.359397 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.350741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46015 (local address 127.23.173.193:40999)
0812 14:27:46.350896 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:46.350938 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:46.351967 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:46.352233 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:46.352238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:46.352502 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:46.352692 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:46.352697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.353386 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.353387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:46.354187 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:46.354190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.354557 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.354558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:46.354673 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.355126 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:46.355134 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:46.356505 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:46.357889 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:46.357897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:46.357899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:46.358175 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:46.358378 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:46.358379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:46.358379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:46.358495 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:46.358738 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:46.358745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:46.358904 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 14:27:46.359017 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:27:46.359186 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260812 14:27:46.359972 1521 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46015
I20260812 14:27:46.360075 1521 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:46.360273 1521 heartbeater.cc:511] Master 127.23.173.254:46015 requested a full tablet report, sending...
I20260812 14:27:46.360859 1191 ts_manager.cc:194] Registered new tserver with Master: a0ccb0837b6e4ce58b575439459ba956 (127.23.173.193:33161)
I20260812 14:27:46.361732 1191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:40999
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:27:46.386723 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46015
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:46.471318 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:46.471534 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:46.471565 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:46.472761 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:46.472810 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:46.472836 1526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:46.474359 1526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:46.474635 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:46.474866 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:46.480103 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:46.480103 1540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.480329 1526 server_base.cc:1034] running on GCE node
W20260812 14:27:46.480155 1535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.480554 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:46.480815 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:46.481988 1526 hybrid_clock.cc:648] HybridClock initialized: now 1786544866481975 us; error 30 us; skew 500 ppm
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.484735 1526 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:27:46.485163 1526 webserver.cc:460] Webserver started at http://127.23.173.194:38653/ using document root <none> and password file <none>
I20260812 14:27:46.485365 1526 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:46.485406 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:46.485497 1526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:46.486517 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "75947b05f227448f94a2532899ded53b"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "9008081e4b2382f6d83fe82cbe32d284"
server_key_iv: "96699a12ecc2f6bc9b445d4035dabb90"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.486827 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "75947b05f227448f94a2532899ded53b"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "9008081e4b2382f6d83fe82cbe32d284"
server_key_iv: "96699a12ecc2f6bc9b445d4035dabb90"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.488253 1526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:46.489014 1549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.489192 1526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:46.489283 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "75947b05f227448f94a2532899ded53b"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "9008081e4b2382f6d83fe82cbe32d284"
server_key_iv: "96699a12ecc2f6bc9b445d4035dabb90"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.489343 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:46.503818 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:46.506031 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:46.506163 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:46.506363 1526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:46.506659 1526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:46.506697 1526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.506719 1526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:46.506731 1526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.512455 1526 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:44457
I20260812 14:27:46.512509 1763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:44457 every 8 connection(s)
I20260812 14:27:46.512828 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:46.521977 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.513530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:58681 (local address 127.23.173.254:46015)
0812 14:27:46.513588 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.513590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.513767 (+ 177us) server_negotiation.cc:407] Connection header received
0812 14:27:46.514763 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.514764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.514778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.514833 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.515519 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.516078 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.517066 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.517252 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.517308 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.519440 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.519442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.519442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.519450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.520762 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.521164 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.521166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.521166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.521230 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.521476 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.521476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.521477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.521585 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.521632 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.521844 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.521911 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:27:46.522220 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.513485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46015 (local address 127.23.173.194:58681)
0812 14:27:46.513636 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:46.513679 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:46.514640 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:46.514916 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:46.514921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:46.515195 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:46.515418 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:46.515423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.516171 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.516172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:46.516974 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:46.516976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.517324 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.517325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:46.517450 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.517929 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:46.517938 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:46.519337 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:46.520849 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:46.520856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:46.520858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:46.521096 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:46.521317 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:46.521318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:46.521318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:46.521426 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:46.521630 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:46.521633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:46.521750 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 14:27:46.521828 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:27:46.521954 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260812 14:27:46.522723 1764 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46015
I20260812 14:27:46.522836 1764 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:46.522903 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 1526
I20260812 14:27:46.523022 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:46.523044 1764 heartbeater.cc:511] Master 127.23.173.254:46015 requested a full tablet report, sending...
I20260812 14:27:46.523212 24247 external_mini_cluster.cc:1428] Setting key ba2222346109a8dcf215c2069418f8ae
I20260812 14:27:46.523425 1191 ts_manager.cc:194] Registered new tserver with Master: 75947b05f227448f94a2532899ded53b (127.23.173.194:44457)
I20260812 14:27:46.523934 1191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:58681
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:27:46.550709 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46015
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:46.633648 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:46.633863 1770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:46.633896 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:46.635104 1770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:46.635151 1770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:46.635188 1770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:46.636767 1770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:46.637009 1770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:46.637226 1770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:46.642294 1784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:46.642313 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:46.642294 1782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:46.642616 1770 server_base.cc:1034] running on GCE node
I20260812 14:27:46.642851 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:46.643102 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:46.644258 1770 hybrid_clock.cc:648] HybridClock initialized: now 1786544866644242 us; error 32 us; skew 500 ppm
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.646888 1770 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:27:46.647333 1770 webserver.cc:460] Webserver started at http://127.23.173.195:43591/ using document root <none> and password file <none>
I20260812 14:27:46.647536 1770 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:46.647586 1770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:46.647753 1770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:46.648846 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "24542312162140498a19e8f4059db923"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "e104acb83c15268440cc48825e5e1a59"
server_key_iv: "92323311525f471c39c8b1d42f9503ef"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.649211 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "24542312162140498a19e8f4059db923"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "e104acb83c15268440cc48825e5e1a59"
server_key_iv: "92323311525f471c39c8b1d42f9503ef"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.650525 1770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:46.651227 1793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.651383 1770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:46.651469 1770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "24542312162140498a19e8f4059db923"
format_stamp: "Formatted at 2026-08-12 14:27:46 on dist-test-slave-1l3l"
server_key: "e104acb83c15268440cc48825e5e1a59"
server_key_iv: "92323311525f471c39c8b1d42f9503ef"
server_key_version: "encryptionkey@0"
I20260812 14:27:46.651540 1770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:46.664000 1770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:46.666090 1770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:46.666245 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:46.666517 1770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:46.666850 1770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:46.666885 1770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.666915 1770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:46.666932 1770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:46.672778 1770 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:40149
I20260812 14:27:46.672825 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:40149 every 8 connection(s)
I20260812 14:27:46.673151 1770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:46.676468 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 1770
I20260812 14:27:46.676599 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:46.676790 24247 external_mini_cluster.cc:1428] Setting key cb2e8692163f0cae6ae662a874743073
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:27:46.682339 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.673861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:53565 (local address 127.23.173.254:46015)
0812 14:27:46.673941 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.673943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.674125 (+ 182us) server_negotiation.cc:407] Connection header received
0812 14:27:46.675079 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.675080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.675091 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.675141 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.675786 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.676301 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.677495 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.677737 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.677792 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.680130 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.680131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.680132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.680138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.681270 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.681637 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.681637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.681638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.681682 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.681866 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.681867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.681867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.681963 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.682006 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.682216 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.682274 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:27:46.682580 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.673824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46015 (local address 127.23.173.195:53565)
0812 14:27:46.673995 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:46.674033 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:46.674987 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:46.675199 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:46.675202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:46.675485 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:46.675688 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:46.675693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.676396 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.676397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:46.677421 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:46.677424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.677811 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.677812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:46.677944 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.678514 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:46.678526 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:46.680038 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:46.681336 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:46.681343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:46.681344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:46.681581 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:46.681728 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:46.681728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:46.681729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:46.681823 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:46.682006 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:46.682009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:46.682125 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 14:27:46.682202 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:27:46.682324 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260812 14:27:46.683147 1998 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46015
I20260812 14:27:46.683243 1998 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:46.683394 1998 heartbeater.cc:511] Master 127.23.173.254:46015 requested a full tablet report, sending...
I20260812 14:27:46.683827 1191 ts_manager.cc:194] Registered new tserver with Master: 24542312162140498a19e8f4059db923 (127.23.173.195:40149)
I20260812 14:27:46.684222 1191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:53565
I20260812 14:27:46.688506 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/test-admin.keytab.
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.706065 24247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 14:27:46.707784 2010 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:46.707011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.23.173.193:33161)
0812 14:27:46.707188 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.707194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.707204 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:46.707232 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.707234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.707248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.707315 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.707651 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53312: BlockingRecv error: recv got EOF from 127.0.0.1:53312 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
W20260812 14:27:46.707854 24247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.708775 24247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.709625 24247 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:46.717176 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.710140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.23.173.193:33161)
0812 14:27:46.710192 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.710193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.710247 (+ 54us) server_negotiation.cc:407] Connection header received
0812 14:27:46.710291 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.710291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.710301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.710349 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.710685 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.711438 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.712165 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.712493 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.712551 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.714514 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.714523 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.714523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.714530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.716104 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.716416 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.716418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.716418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.716470 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.716656 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.716658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.716658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.716816 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.716938 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.716998 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.717064 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 14:27:46.718971 2010 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:46.718383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.23.173.193:33161)
0812 14:27:46.718422 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.718423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.718514 (+ 91us) server_negotiation.cc:407] Connection header received
0812 14:27:46.718543 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.718543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.718554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.718606 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.718935 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53322: BlockingRecv error: recv got EOF from 127.0.0.1:53322 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:27:46.719077 24247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.719988 24247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.720708 24247 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:46.727162 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.721240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.23.173.193:33161)
0812 14:27:46.721314 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.721316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.721435 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:27:46.721493 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.721494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.721504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.721548 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.721884 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.722352 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.722996 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.723194 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.723247 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.725157 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.725158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.725158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.725165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.726223 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.726517 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.726518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.726518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.726568 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.726722 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.726723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.726723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.726813 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.726902 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.726992 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.727069 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 14:27:46.728554 2010 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:46.728155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.23.173.193:33161)
0812 14:27:46.728192 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.728193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.728203 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:46.728246 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.728247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.728258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.728299 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.728511 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53346: BlockingRecv error: recv got EOF from 127.0.0.1:53346 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 14:27:46.728715 24247 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544840008271-24247-0/krb5kdc/krb5cc)
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.729627 24247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:46.730556 24247 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[1122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.193@KRBTEST.COM
I20260812 14:27:46.737224 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:46.731058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.23.173.193:33161)
0812 14:27:46.731108 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:46.731110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:46.731142 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:27:46.731212 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:46.731213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:46.731222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:46.731266 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:46.731615 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.732142 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.732813 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:46.733016 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:46.733068 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:46.734917 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:46.734918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:46.734918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:46.734926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:46.736234 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.736542 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.736543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.736543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.736593 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:46.736755 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:46.736756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:46.736756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:46.736850 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:46.736952 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:46.737053 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:27:46.737127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:27:46.737897 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 1291
I20260812 14:27:46.749397 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 1526
I20260812 14:27:46.770630 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 1770
I20260812 14:27:46.783354 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 1143
2026-08-12T14:27:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2064 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[2036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[2036](info): set up 2 sockets
Aug 12 14:27:46 dist-test-slave-1l3l krb5kdc[2036](info): commencing operation
krb5kdc: starting...
W20260812 14:27:47.977798 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab.
Entry for principal oryx/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab.
I20260812 14:27:48.104104 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46677
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab
--principal=oryx/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:48.201491 2062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:48.201689 2062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:48.201710 2062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:48.202847 2062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:48.202890 2062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:48.202903 2062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:48.202914 2062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:48.202924 2062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:48.204424 2062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab
--principal=oryx/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:48.204658 2062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:48.204880 2062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:48.209688 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.209703 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.209895 2062 server_base.cc:1034] running on GCE node
W20260812 14:27:48.209939 2073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.210161 2062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:48.210445 2062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:48.211596 2062 hybrid_clock.cc:648] HybridClock initialized: now 1786544868211583 us; error 30 us; skew 500 ppm
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:48.213989 2062 init.cc:377] Logged in from keytab as oryx/127.23.173.254@KRBTEST.COM (short username oryx)
I20260812 14:27:48.214370 2062 webserver.cc:460] Webserver started at http://127.23.173.254:42339/ using document root <none> and password file <none>
I20260812 14:27:48.214552 2062 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:48.214592 2062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:48.214684 2062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:48.215673 2062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "9fc8af0aba90482090c23064205f5fb3"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "89a8cc6bd49f052ec855db0a8947d177"
server_key_iv: "d5b8608b4fecca4d7698a480dc20858d"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.216080 2062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "9fc8af0aba90482090c23064205f5fb3"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "89a8cc6bd49f052ec855db0a8947d177"
server_key_iv: "d5b8608b4fecca4d7698a480dc20858d"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.217267 2062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:48.217935 2086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.218117 2062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:48.218209 2062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "9fc8af0aba90482090c23064205f5fb3"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "89a8cc6bd49f052ec855db0a8947d177"
server_key_iv: "d5b8608b4fecca4d7698a480dc20858d"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.218263 2062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:48.234558 2062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:48.236754 2062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:48.236869 2062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:48.240636 2062 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46677
I20260812 14:27:48.240677 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46677 every 8 connection(s)
I20260812 14:27:48.241074 2062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:48.242100 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:48.244465 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Bootstrap starting.
I20260812 14:27:48.245021 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:48.245260 2185 log.cc:824] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:48.245945 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: No bootstrap required, opened a new log
I20260812 14:27:48.247545 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } }
I20260812 14:27:48.247689 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:48.247735 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc8af0aba90482090c23064205f5fb3, State: Initialized, Role: FOLLOWER
I20260812 14:27:48.247848 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } }
I20260812 14:27:48.247924 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:48.247952 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:48.247989 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:48.248785 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } }
I20260812 14:27:48.248873 2185 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc8af0aba90482090c23064205f5fb3; no voters:
I20260812 14:27:48.249002 2185 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:48.249049 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:48.249182 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 9fc8af0aba90482090c23064205f5fb3, State: Running, Role: LEADER
I20260812 14:27:48.249265 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:48.249284 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 2062
I20260812 14:27:48.249290 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } }
I20260812 14:27:48.249382 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:27:48.249580 24247 external_mini_cluster.cc:1428] Setting key a382e641feb52f04e27ff120a36dfb5d
I20260812 14:27:48.249684 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } }
I20260812 14:27:48.249761 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:48.249707 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc8af0aba90482090c23064205f5fb3. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } }
I20260812 14:27:48.249871 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:48.249989 2213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:48.250524 2213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:48.252075 2213 catalog_manager.cc:1347] Generated new cluster ID: 9714c2194a8846d4aff3f2ec0a858e06
I20260812 14:27:48.252170 2213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:27:48.261878 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.250424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.23.173.254:46677)
0812 14:27:48.250626 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.250636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.250792 (+ 156us) server_negotiation.cc:407] Connection header received
0812 14:27:48.251146 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.251154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.251222 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.251378 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.252154 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.253542 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.254640 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.255269 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.255376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.258179 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.258184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.258184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.258198 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.260254 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.260629 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.260632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.260632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.260743 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.260913 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.260914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.260914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.261264 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.261421 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.261490 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.261636 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260812 14:27:48.273227 2213 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:48.273962 2213 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:48.277549 2213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Generated new TSK 0
I20260812 14:27:48.277737 2213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab.
Entry for principal oryx/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab.
I20260812 14:27:48.307969 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46677
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:48.392294 2237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:48.392498 2237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:48.392520 2237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:48.393668 2237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:48.393709 2237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:48.393734 2237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:48.395176 2237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:48.395426 2237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:48.395645 2237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:48.400607 2237 server_base.cc:1034] running on GCE node
W20260812 14:27:48.400589 2242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.400589 2244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.400573 2247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.400967 2237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:48.401197 2237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:48.402334 2237 hybrid_clock.cc:648] HybridClock initialized: now 1786544868402325 us; error 33 us; skew 500 ppm
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:48.404805 2237 init.cc:377] Logged in from keytab as oryx/127.23.173.193@KRBTEST.COM (short username oryx)
I20260812 14:27:48.405221 2237 webserver.cc:460] Webserver started at http://127.23.173.193:43269/ using document root <none> and password file <none>
I20260812 14:27:48.405433 2237 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:48.405480 2237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:48.405602 2237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:48.406694 2237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "da6868b5df5d451c998fcd4d73180f49"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "5ff7e126efb948e00ed25fc9d0b6f755"
server_key_iv: "70c323f45d0163934649f906c17ec640"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.407029 2237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "da6868b5df5d451c998fcd4d73180f49"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "5ff7e126efb948e00ed25fc9d0b6f755"
server_key_iv: "70c323f45d0163934649f906c17ec640"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.408309 2237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:48.408973 2259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.409121 2237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:48.409200 2237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "da6868b5df5d451c998fcd4d73180f49"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "5ff7e126efb948e00ed25fc9d0b6f755"
server_key_iv: "70c323f45d0163934649f906c17ec640"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.409258 2237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:48.438429 2237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:48.440603 2237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:48.440764 2237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:48.441020 2237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:48.441344 2237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:48.441378 2237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.441408 2237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:48.441425 2237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.447209 2237 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:43703
I20260812 14:27:48.447274 2453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:43703 every 8 connection(s)
I20260812 14:27:48.447599 2237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:27:48.454413 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 2237
I20260812 14:27:48.454519 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:27:48.454686 24247 external_mini_cluster.cc:1428] Setting key 75ddcb0cc59362ca24f875e3fa9cdd7f
I20260812 14:27:48.457006 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.448370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51195 (local address 127.23.173.254:46677)
0812 14:27:48.448421 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.448423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.448663 (+ 240us) server_negotiation.cc:407] Connection header received
0812 14:27:48.449165 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.449167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.449181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.449255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.450286 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.450830 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.451863 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.452072 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.452122 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.454268 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.454269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.454270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.454277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.455683 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.456100 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.456102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.456102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.456156 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.456384 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.456385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.456385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.456499 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.456555 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.456824 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.456914 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:27:48.457247 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.448333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:51195)
0812 14:27:48.448511 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.448556 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.449045 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.449325 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.449329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.449966 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.450171 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.450177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.450922 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.450923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.451797 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.451800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.452130 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.452131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.452232 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.452744 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.452753 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.454154 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.455788 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.455796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.455798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.456032 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.456211 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.456212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.456212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.456323 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.456557 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.456564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.456703 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.456794 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.456973 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260812 14:27:48.457731 2457 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46677
I20260812 14:27:48.457834 2457 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:48.458021 2457 heartbeater.cc:511] Master 127.23.173.254:46677 requested a full tablet report, sending...
I20260812 14:27:48.458554 2121 ts_manager.cc:194] Registered new tserver with Master: da6868b5df5d451c998fcd4d73180f49 (127.23.173.193:43703)
I20260812 14:27:48.459167 2121 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:51195
WARNING: no policy specified for oryx/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab.
Entry for principal oryx/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab.
I20260812 14:27:48.482499 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46677
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:48.564905 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:48.565140 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:48.565165 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:48.566303 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:48.566339 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:48.566365 2462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:48.567869 2462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:48.568084 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:48.568280 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:48.573294 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.573307 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.573375 2462 server_base.cc:1034] running on GCE node
W20260812 14:27:48.573303 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.573773 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:48.574028 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:48.575187 2462 hybrid_clock.cc:648] HybridClock initialized: now 1786544868575166 us; error 36 us; skew 500 ppm
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:48.577829 2462 init.cc:377] Logged in from keytab as oryx/127.23.173.194@KRBTEST.COM (short username oryx)
I20260812 14:27:48.578293 2462 webserver.cc:460] Webserver started at http://127.23.173.194:43027/ using document root <none> and password file <none>
I20260812 14:27:48.578511 2462 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:48.578554 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:48.578645 2462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:48.579674 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "8002180f25e6495c885ec1a89793c164"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "3bf5c5fe9a0c350ffd9f6818762fbc24"
server_key_iv: "a7606ed76d813bdb586a11cea261ee1b"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.580046 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "8002180f25e6495c885ec1a89793c164"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "3bf5c5fe9a0c350ffd9f6818762fbc24"
server_key_iv: "a7606ed76d813bdb586a11cea261ee1b"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.581331 2462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:48.582167 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.582360 2462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:48.582489 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "8002180f25e6495c885ec1a89793c164"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "3bf5c5fe9a0c350ffd9f6818762fbc24"
server_key_iv: "a7606ed76d813bdb586a11cea261ee1b"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.582551 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:48.604272 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:48.606350 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:48.606482 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:48.606709 2462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:48.606997 2462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:48.607030 2462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.607053 2462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:48.607067 2462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.612908 2462 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:36641
I20260812 14:27:48.613044 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:36641 every 8 connection(s)
I20260812 14:27:48.613322 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:27:48.618280 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 2462
I20260812 14:27:48.618374 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:27:48.618614 24247 external_mini_cluster.cc:1428] Setting key 11dfefd4b0261f25d7b542325c05960e
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:27:48.623301 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.614091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:39833 (local address 127.23.173.254:46677)
0812 14:27:48.614150 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.614152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.614369 (+ 217us) server_negotiation.cc:407] Connection header received
0812 14:27:48.614884 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.614886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.614898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.614983 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.615962 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.616454 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.617527 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.617731 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.617796 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.620534 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.620535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.620536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.620544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.621935 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.622338 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.622340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.622340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.622408 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.622646 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.622646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.622647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.622764 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.622822 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.623112 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.623208 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:27:48.623493 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.614057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:39833)
0812 14:27:48.614241 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.614286 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.614780 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.615044 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.615047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.615633 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.615867 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.615873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.616560 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.616561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.617415 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.617418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.617827 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.617829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.617969 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.618619 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.618630 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.620433 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.622020 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.622029 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.622032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.622266 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.622467 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.622468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.622469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.622587 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.622823 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.622831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.622988 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.623081 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.623254 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260812 14:27:48.624154 2671 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46677
I20260812 14:27:48.624285 2671 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:48.624522 2671 heartbeater.cc:511] Master 127.23.173.254:46677 requested a full tablet report, sending...
I20260812 14:27:48.624961 2121 ts_manager.cc:194] Registered new tserver with Master: 8002180f25e6495c885ec1a89793c164 (127.23.173.194:36641)
I20260812 14:27:48.625339 2121 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:39833
WARNING: no policy specified for oryx/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab.
Entry for principal oryx/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab.
I20260812 14:27:48.648201 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46677
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:48.734836 2680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:48.735056 2680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:48.735081 2680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:48.736348 2680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:48.736392 2680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:48.736418 2680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:48.737884 2680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:48.738093 2680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:48.738294 2680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:48.743359 2693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.743352 2690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.743400 2691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.743619 2680 server_base.cc:1034] running on GCE node
I20260812 14:27:48.743968 2680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:48.744282 2680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:48.745443 2680 hybrid_clock.cc:648] HybridClock initialized: now 1786544868745424 us; error 36 us; skew 500 ppm
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:48.748118 2680 init.cc:377] Logged in from keytab as oryx/127.23.173.195@KRBTEST.COM (short username oryx)
I20260812 14:27:48.748622 2680 webserver.cc:460] Webserver started at http://127.23.173.195:34453/ using document root <none> and password file <none>
I20260812 14:27:48.748919 2680 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:48.748977 2680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:48.749138 2680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:48.750314 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "aeb496dd356118a6213b9c43b78ddea1"
server_key_iv: "9fb814e8dfc6b20c06bce1b23ca0fca3"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.750674 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "aeb496dd356118a6213b9c43b78ddea1"
server_key_iv: "9fb814e8dfc6b20c06bce1b23ca0fca3"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.752187 2680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:48.753058 2705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.753288 2680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:48.753381 2680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "aeb496dd356118a6213b9c43b78ddea1"
server_key_iv: "9fb814e8dfc6b20c06bce1b23ca0fca3"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.753448 2680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:48.768246 2680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:48.770183 2680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:48.770334 2680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:48.770571 2680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:48.770879 2680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:48.770915 2680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.770946 2680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:48.770970 2680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.777045 2680 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:35449
I20260812 14:27:48.777108 2912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:35449 every 8 connection(s)
I20260812 14:27:48.777487 2680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:27:48.784593 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 2680
I20260812 14:27:48.784683 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:27:48.784865 24247 external_mini_cluster.cc:1428] Setting key 849ebcf71f4b328c0b11b6699da7f48b
I20260812 14:27:48.787134 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.778286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:45823 (local address 127.23.173.254:46677)
0812 14:27:48.778330 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.778332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.778531 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:27:48.779052 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.779053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.779065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.779115 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.780159 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.780676 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.781730 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.781934 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.782006 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.784217 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.784219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.784219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.784226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.785796 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.786170 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.786171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.786172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.786235 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.786472 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.786472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.786473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.786598 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.786663 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.786953 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.787045 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:27:48.787313 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.778250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:45823)
0812 14:27:48.778407 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.778446 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.778951 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.779172 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.779176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.779829 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.780047 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.780054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.780770 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.780771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.781629 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.781632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.782030 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.782031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.782145 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.782662 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.782670 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.784115 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.785866 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.785874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.785876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.786108 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.786301 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.786302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.786302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.786424 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.786677 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.786685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.786825 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.786920 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.787088 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260812 14:27:48.787791 2913 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46677
I20260812 14:27:48.787881 2913 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:48.788081 2913 heartbeater.cc:511] Master 127.23.173.254:46677 requested a full tablet report, sending...
I20260812 14:27:48.788434 2121 ts_manager.cc:194] Registered new tserver with Master: 5ea4495ded204ab3a688d5d1cd9b0ce0 (127.23.173.195:35449)
I20260812 14:27:48.788784 2121 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:45823
I20260812 14:27:48.797056 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.173.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.253.keytab.
Entry for principal oryx/127.23.173.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.253.keytab.
I20260812 14:27:48.827033 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:38203
--webserver_interface=127.23.173.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.253.keytab
--principal=oryx/127.23.173.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46677,127.23.173.253:38203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:48.910640 2923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:48.910857 2923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:48.910889 2923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:48.912138 2923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:48.912184 2923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:48.912204 2923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:48.912220 2923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:48.912236 2923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:48.913811 2923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46677,127.23.173.253:38203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.253.keytab
--principal=oryx/127.23.173.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:38203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
W20260812 14:27:48.913930 2923 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.173.254:46677,127.23.173.253:38203'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 14:27:48.914073 2923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:48.914283 2923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:48.919210 2932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.919178 2930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:48.919203 2935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:48.919242 2923 server_base.cc:1034] running on GCE node
I20260812 14:27:48.919683 2923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:48.920042 2923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:48.921187 2923 hybrid_clock.cc:648] HybridClock initialized: now 1786544868921172 us; error 26 us; skew 500 ppm
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:48.923799 2923 init.cc:377] Logged in from keytab as oryx/127.23.173.253@KRBTEST.COM (short username oryx)
I20260812 14:27:48.924422 2923 webserver.cc:460] Webserver started at http://127.23.173.253:42371/ using document root <none> and password file <none>
I20260812 14:27:48.924669 2923 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:48.924716 2923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:48.924830 2923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:48.925930 2923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/instance:
uuid: "c1367ca601474caa881012adc5ea6714"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "db6bf0277185168af48a3245190ac864"
server_key_iv: "896cdadec3383d6243e67bb7edcf5d71"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.926268 2923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal/instance:
uuid: "c1367ca601474caa881012adc5ea6714"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "db6bf0277185168af48a3245190ac864"
server_key_iv: "896cdadec3383d6243e67bb7edcf5d71"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.927552 2923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:48.928304 2954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.928470 2923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:48.928561 2923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
uuid: "c1367ca601474caa881012adc5ea6714"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "db6bf0277185168af48a3245190ac864"
server_key_iv: "896cdadec3383d6243e67bb7edcf5d71"
server_key_version: "encryptionkey@0"
I20260812 14:27:48.928618 2923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:48.952380 2923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:48.954258 2923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:48.954377 2923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:48.958405 2923 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.253:38203
I20260812 14:27:48.958441 3052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.253:38203 every 8 connection(s)
I20260812 14:27:48.958823 2923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
I20260812 14:27:48.959880 3053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:48.961437 3053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:48.962944 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 2923
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.253@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
I20260812 14:27:48.970896 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.962140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.23.173.254:46677)
0812 14:27:48.962191 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.962193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.962400 (+ 207us) server_negotiation.cc:407] Connection header received
0812 14:27:48.962862 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.962864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.962878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.962976 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.963827 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.964433 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.965444 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.965648 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.965705 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.967924 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.967926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.967926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.967934 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.969350 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.969839 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.969840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.969841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.969924 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.970216 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.970217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.970218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.970369 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.970463 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.970722 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.970805 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:27:48.971124 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.962108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49380)
0812 14:27:48.962288 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.962330 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.962768 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.963036 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.963041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.963346 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.963698 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.963741 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.964530 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.964531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.965356 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.965359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.965722 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.965723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.965842 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.966354 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.966362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.967843 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.969441 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.969447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.969449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.969749 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.970030 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.970031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.970032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.970143 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.970465 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.970472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.970596 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.970684 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.970839 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260812 14:27:48.971843 3053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:48.971958 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.963750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.23.173.253:38203)
0812 14:27:48.963906 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.963910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.964264 (+ 354us) server_negotiation.cc:407] Connection header received
0812 14:27:48.964291 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.964294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.964316 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.964403 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.964972 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.965695 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.966545 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.966859 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.966931 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.969196 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.969204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.969204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.969211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.970732 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.971068 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.971082 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.971082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.971145 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.971383 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.971384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.971385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.971558 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.971700 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.971820 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.971890 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.253@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
I20260812 14:27:48.977349 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.972102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.0.0.1:59800)
0812 14:27:48.972159 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.972168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.972205 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.972315 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.972315 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.972415 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.972444 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.972445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.973064 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.973065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.973524 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.973526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.973844 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.973845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.973907 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.974203 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.974207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.975268 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.976626 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.976628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.976628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.976782 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.976930 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.976930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.976930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.976973 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.977137 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.977137 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.977169 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.977211 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.977277 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 14:27:48.977401 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.972130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.23.173.253:38203)
0812 14:27:48.972161 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.972162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.972194 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:27:48.972246 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.972246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.972255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.972312 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.972507 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.973000 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.973583 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.973768 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.973813 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.975346 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.975348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.975348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.975354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.976541 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.976842 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.976843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.976843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.976885 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.977017 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.977017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.977017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.977098 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.977139 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.977227 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.977317 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":15}
I20260812 14:27:48.978631 3053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Bootstrap starting.
I20260812 14:27:48.979453 3053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:48.979763 3053 log.cc:824] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:48.980459 3053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: No bootstrap required, opened a new log
I20260812 14:27:48.981863 3053 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } }
I20260812 14:27:48.981983 3053 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:48.982014 3053 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1367ca601474caa881012adc5ea6714, State: Initialized, Role: FOLLOWER
I20260812 14:27:48.982134 3053 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } }
I20260812 14:27:48.982410 3053 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:48.982394 3062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } } }
I20260812 14:27:48.982489 3062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:27:48.984094 3080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:48.984144 3080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:48.985771 2923 master_runner.cc:428] Detected that this master c1367ca601474caa881012adc5ea6714 is joining an existing cluster
I20260812 14:27:48.985864 2923 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.173.253:38203
I20260812 14:27:48.993312 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.987323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.0.0.1:59802)
0812 14:27:48.987431 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.987466 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.987522 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.987634 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.987635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.987850 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.987908 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.987910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.988771 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.988772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.989399 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.989401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.989759 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.989759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.989813 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.990245 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.990250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.990633 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.992383 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.992384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.992384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.992545 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.992706 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.992706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.992706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.992759 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.992962 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.992962 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.993000 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.993080 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.993192 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":50,"thread_start_us":55,"threads_started":1}
I20260812 14:27:48.993379 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.987051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.23.173.253:38203)
0812 14:27:48.987083 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.987085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.987524 (+ 439us) server_negotiation.cc:407] Connection header received
0812 14:27:48.987572 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.987573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.987582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.987626 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.987983 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.988687 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.989503 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.989691 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.989753 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.990711 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.990712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.990713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.990719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.992311 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.992604 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.992604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.992605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.992654 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.992808 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.992809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.992809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.992910 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.992969 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.993107 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.993254 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":18}
I20260812 14:27:48.993841 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.986963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.23.173.254:46677)
0812 14:27:48.987006 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.987008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.987194 (+ 186us) server_negotiation.cc:407] Connection header received
0812 14:27:48.987286 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.987287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.987298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.987327 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.987828 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.988385 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.989750 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.990048 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.990092 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.991099 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:48.991100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:48.991100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:48.991108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:48.992636 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.993039 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.993040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.993040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.993101 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:48.993295 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:48.993296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:48.993296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:48.993422 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:48.993470 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:48.993674 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 14:27:48.993758 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:27:48.994240 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.986882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49394)
0812 14:27:48.987131 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.987142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.987217 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.987388 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.987389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.987604 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.987757 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.987760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.988776 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.988777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.989648 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.989651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.990168 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.990169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.990242 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.990608 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.990613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:48.991016 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:48.992727 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.992728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.992729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.992940 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.993171 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:48.993172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:48.993172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:48.993228 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:48.993519 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:48.993520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:48.993557 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:27:48.993909 (+ 352us) client_negotiation.cc:240] Negotiation successful
0812 14:27:48.993997 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
Aug 12 14:27:48 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.254@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
I20260812 14:27:49.003072 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.996374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.23.173.253:38203)
0812 14:27:48.996408 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:48.996409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:48.996559 (+ 150us) server_negotiation.cc:407] Connection header received
0812 14:27:48.996618 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:48.996618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:48.996628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:48.996690 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:48.997041 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.997754 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.998487 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.998784 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.998846 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.000845 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.000847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.000847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.000853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.002055 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.002459 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.002460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.002460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.002500 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.002665 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.002665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.002665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.002749 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.002790 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.002964 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.003013 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:49.003213 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:48.996309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.0.0.1:59804)
0812 14:27:48.996430 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:48.996501 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:48.996564 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:48.996694 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:48.996696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:48.996931 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:48.996981 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.996983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.997837 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.997838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.998407 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:48.998409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:48.998844 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:48.998845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:48.998897 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:48.999325 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:48.999329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.000761 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.002127 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.002135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.002138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.002394 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.002550 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.002551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.002551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.002615 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.002797 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.002803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.002855 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.002938 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.003052 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260812 14:27:49.003810 2121 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.173.253:38203: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c1367ca601474caa881012adc5ea6714"
member_type: NON_VOTER
last_known_addr {
host: "127.23.173.253"
port: 38203
}
attrs {
promote: true
}
}
dest_uuid: "9fc8af0aba90482090c23064205f5fb3"
cas_config_opid_index: -1
I20260812 14:27:49.004166 2121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } }
I20260812 14:27:49.004675 3098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c1367ca601474caa881012adc5ea6714 (127.23.173.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } }
I20260812 14:27:49.004976 3007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:49.005777 3098 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.173.253:38203
I20260812 14:27:49.005885 3007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Refusing update from remote peer 9fc8af0aba90482090c23064205f5fb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 14:27:49.005992 3096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } }
I20260812 14:27:49.006062 3096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:49.006076 2923 master.cc:537] Master@127.23.173.253:38203 shutting down...
I20260812 14:27:49.006690 3096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:27:49.007356 2087 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 14:27:49.008143 2087 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 -> Peer c1367ca601474caa881012adc5ea6714 (127.23.173.253:38203): Couldn't send request to peer c1367ca601474caa881012adc5ea6714. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:27:49.009151 2923 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:49.009241 2923 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:49.009291 2923 tablet_replica.cc:330] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: stopping tablet replica
I20260812 14:27:49.022168 2923 master.cc:552] Master@127.23.173.253:38203 shutdown complete.
I20260812 14:27:49.022214 2923 master_runner.cc:305] Clearing existing system tablet
I20260812 14:27:49.022331 2923 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.022373 2923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.023239 2923 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:27:49.023941 3105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.027081 2923 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.027233 2923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 14:27:49.027284 2923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
uuid: "c1367ca601474caa881012adc5ea6714"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "db6bf0277185168af48a3245190ac864"
server_key_iv: "896cdadec3383d6243e67bb7edcf5d71"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.028312 2923 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:49.029873 2923 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 14:27:49.029994 2923 log.cc:1196] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 14:27:49.030233 2923 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Deleting consensus metadata
I20260812 14:27:49.030354 2923 master_runner.cc:315] Copying system tablet from 127.23.173.254:46677
I20260812 14:27:49.031015 2923 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: tablet copy: Beginning tablet copy session from remote peer at address 127.23.173.254:46677
I20260812 14:27:49.037500 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.031371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49406)
0812 14:27:49.031557 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.031568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.031664 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.031909 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.031910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.032146 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.032229 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.032233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.033161 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.033161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.033708 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.033709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.034114 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.034114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.034157 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.034613 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.034617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.034908 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.036665 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.036666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.036667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.036827 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.037003 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.037003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.037004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.037049 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.037271 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.037271 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.037307 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.037349 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.037420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260812 14:27:49.037549 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.031437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.23.173.254:46677)
0812 14:27:49.031501 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.031503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.031635 (+ 132us) server_negotiation.cc:407] Connection header received
0812 14:27:49.031794 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.031795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.031810 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.031885 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.032304 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.033090 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.033776 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.034047 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.034110 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.034995 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.034996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.034997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.035005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.036591 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.036890 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.036891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.036891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.036949 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.037096 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.037097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.037097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.037221 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.037284 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.037373 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.037448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:27:49.037878 2156 tablet_copy_service.cc:140] P 9fc8af0aba90482090c23064205f5fb3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c1367ca601474caa881012adc5ea6714 ({username='oryx', principal='oryx/127.23.173.253@KRBTEST.COM'} at 127.0.0.1:49406)
I20260812 14:27:49.038028 2156 tablet_copy_service.cc:161] P 9fc8af0aba90482090c23064205f5fb3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c1367ca601474caa881012adc5ea6714 at {username='oryx', principal='oryx/127.23.173.253@KRBTEST.COM'} at 127.0.0.1:49406: session id = c1367ca601474caa881012adc5ea6714-00000000000000000000000000000000
I20260812 14:27:49.038758 2156 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:27:49.039335 2923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 14:27:49.040391 2923 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: tablet copy: Starting download of 0 data blocks...
I20260812 14:27:49.040539 2923 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: tablet copy: Starting download of 1 WAL segments...
I20260812 14:27:49.041508 2923 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:27:49.042338 2156 tablet_copy_service.cc:342] P 9fc8af0aba90482090c23064205f5fb3: Request end of tablet copy session c1367ca601474caa881012adc5ea6714-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.173.253@KRBTEST.COM'} at 127.0.0.1:49406
I20260812 14:27:49.042397 2156 tablet_copy_service.cc:433] P 9fc8af0aba90482090c23064205f5fb3: ending tablet copy session c1367ca601474caa881012adc5ea6714-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c1367ca601474caa881012adc5ea6714
I20260812 14:27:49.046516 2923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:27:49.047833 2923 server_base.cc:1034] running on GCE node
W20260812 14:27:49.047860 3130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.047879 3129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.048138 3132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.048373 2923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:49.048543 2923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:49.049686 2923 hybrid_clock.cc:648] HybridClock initialized: now 1786544869049681 us; error 37 us; skew 500 ppm
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:49.051038 2923 init.cc:377] Logged in from keytab as oryx/127.23.173.253@KRBTEST.COM (short username oryx)
I20260812 14:27:49.051295 2923 webserver.cc:460] Webserver started at http://127.23.173.253:44375/ using document root <none> and password file <none>
I20260812 14:27:49.051397 2923 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.051431 2923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.052068 2923 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:49.052587 3145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.052753 2923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.052796 2923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
uuid: "c1367ca601474caa881012adc5ea6714"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "db6bf0277185168af48a3245190ac864"
server_key_iv: "896cdadec3383d6243e67bb7edcf5d71"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.052852 2923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.068475 2923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:49.070056 2923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:49.072817 2923 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.253:38203
I20260812 14:27:49.072860 3221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.253:38203 every 8 connection(s)
I20260812 14:27:49.073150 2923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
I20260812 14:27:49.074116 3223 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:27:49.074731 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Bootstrap starting.
I20260812 14:27:49.077633 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:27:49.077847 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: Bootstrap complete.
I20260812 14:27:49.078045 3223 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } }
I20260812 14:27:49.078119 3223 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c1367ca601474caa881012adc5ea6714, State: Initialized, Role: LEARNER
I20260812 14:27:49.078156 3223 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } }
I20260812 14:27:49.078361 3223 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:49.078343 3231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } }
I20260812 14:27:49.080396 3231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:27:49.080502 3250 catalog_manager.cc:1259] Loaded cluster ID: 9714c2194a8846d4aff3f2ec0a858e06
I20260812 14:27:49.080595 3250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: loading cluster ID for follower catalog manager: success
I20260812 14:27:49.081799 3250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: acquiring CA information for follower catalog manager: success
I20260812 14:27:49.082129 3250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:27:49.082628 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.073603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.23.173.253:38203)
0812 14:27:49.073755 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.073757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.073767 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:49.073836 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.073837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.073848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.073904 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.074467 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.075174 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.076170 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.076425 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.076497 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.078001 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.078003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.078003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.078010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.081170 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.081597 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.081598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.081598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.081669 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.081920 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.081921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.081921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.082062 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.082166 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.082464 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.082565 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.23.173.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.252.keytab.
Entry for principal oryx/127.23.173.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.252.keytab.
I20260812 14:27:49.114820 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:32849
--webserver_interface=127.23.173.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.252.keytab
--principal=oryx/127.23.173.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:49.198710 3260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:49.198910 3260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:49.198935 3260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:49.200138 3260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:49.200181 3260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:49.200194 3260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:49.200206 3260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:49.200217 3260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:49.201769 3260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.252.keytab
--principal=oryx/127.23.173.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:32849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:49.202000 3260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:49.202234 3260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:49.206827 3278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.206843 3275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.207165 3260 server_base.cc:1034] running on GCE node
W20260812 14:27:49.206847 3273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.207448 3260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:49.207799 3260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:49.208943 3260 hybrid_clock.cc:648] HybridClock initialized: now 1786544869208933 us; error 29 us; skew 500 ppm
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:49.211441 3260 init.cc:377] Logged in from keytab as oryx/127.23.173.252@KRBTEST.COM (short username oryx)
I20260812 14:27:49.211895 3260 webserver.cc:460] Webserver started at http://127.23.173.252:40451/ using document root <none> and password file <none>
I20260812 14:27:49.212132 3260 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.212182 3260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.212316 3260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:49.213351 3260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/instance:
uuid: "b6a6c4aee9dd48cc966a5ac49e536b26"
format_stamp: "Formatted at 2026-08-12 14:27:49 on dist-test-slave-1l3l"
server_key: "bc7b082357ebc6586fc57f088fac4011"
server_key_iv: "9c13a25b0af5953c54276b6edb9275bc"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.213701 3260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal/instance:
uuid: "b6a6c4aee9dd48cc966a5ac49e536b26"
format_stamp: "Formatted at 2026-08-12 14:27:49 on dist-test-slave-1l3l"
server_key: "bc7b082357ebc6586fc57f088fac4011"
server_key_iv: "9c13a25b0af5953c54276b6edb9275bc"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.214978 3260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:49.215673 3286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.215850 3260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:49.215934 3260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
uuid: "b6a6c4aee9dd48cc966a5ac49e536b26"
format_stamp: "Formatted at 2026-08-12 14:27:49 on dist-test-slave-1l3l"
server_key: "bc7b082357ebc6586fc57f088fac4011"
server_key_iv: "9c13a25b0af5953c54276b6edb9275bc"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.216013 3260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.229745 3260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:49.231935 3260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:49.232187 3260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:49.236064 3260 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.252:32849
I20260812 14:27:49.236095 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.252:32849 every 8 connection(s)
I20260812 14:27:49.236429 3260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
I20260812 14:27:49.237367 3384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:49.238839 3384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:49.240651 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 3260
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.252@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.252@KRBTEST.COM
I20260812 14:27:49.248354 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.239560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.23.173.254:46677)
0812 14:27:49.239613 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.239615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.239728 (+ 113us) server_negotiation.cc:407] Connection header received
0812 14:27:49.240198 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.240199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.240210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.240269 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.241047 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.241727 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.242898 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.243128 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.243184 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.245627 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.245629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.245629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.245636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.246984 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.247393 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.247394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.247394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.247452 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.247737 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.247738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.247738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.247882 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.247957 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.248186 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.248263 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:27:49.248432 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.239432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49422)
0812 14:27:49.239594 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.239637 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.240105 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.240332 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.240335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.240667 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.240937 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.240942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.241819 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.241820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.242807 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.242809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.243218 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.243219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.243349 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.243838 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.243847 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.245542 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.247054 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.247061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.247063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.247324 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.247525 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.247526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.247526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.247653 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.247948 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.247952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.248062 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.248143 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.248271 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260812 14:27:49.248955 3384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:49.249608 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.241338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.23.173.252:32849)
0812 14:27:49.241496 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.241501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.241523 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:27:49.241546 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.241548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.241568 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.241645 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.242351 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.243251 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.243971 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.244389 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.244468 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.246685 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.246693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.246693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.246700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.248399 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.248765 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.248768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.248768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.248827 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.249078 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.249079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.249079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.249243 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.249386 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.249432 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.249502 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.252@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
I20260812 14:27:49.254755 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.249193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.0.0.1:59826)
0812 14:27:49.249235 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.249246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.249281 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.249417 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.249418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.249538 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.249580 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.249581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.250311 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.250312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.250825 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.250827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.251215 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.251216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.251283 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.251597 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.251601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.252632 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.253916 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.253917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.253918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.254065 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.254230 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.254231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.254231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.254283 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.254485 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.254486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.254524 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.254584 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.254668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:27:49.254765 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.249282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.23.173.253:38203)
0812 14:27:49.249325 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.249327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.249333 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:27:49.249356 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.249357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.249365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.249408 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.249648 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.250247 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.250899 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.251154 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.251208 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.252722 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.252723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.252723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.252730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.253843 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.254122 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.254124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.254124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.254179 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.254333 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.254333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.254334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.254433 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.254485 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.254613 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.254684 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:27:49.255191 3384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 32849 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.252@KRBTEST.COM for oryx/127.23.173.252@KRBTEST.COM
I20260812 14:27:49.260288 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.255401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.0.0.1:58708)
0812 14:27:49.255441 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.255451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.255489 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.255601 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.255601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.255696 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.255751 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.255753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.256341 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.256342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.256775 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.256776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.257075 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.257076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.257126 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.257393 (+ 267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.257396 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.258363 (+ 967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.259498 (+ 1135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.259499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.259499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.259645 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.259838 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.259839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.259839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.259893 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.260085 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.260085 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.260117 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.260162 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.260233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 14:27:49.260363 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.255462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.23.173.252:32849)
0812 14:27:49.255492 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.255493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.255500 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:49.255547 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.255547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.255555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.255589 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.255806 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.256275 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.256826 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.257009 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.257053 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.258430 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.258432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.258432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.258438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.259434 (+ 996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.259699 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.259699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.259700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.259783 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.259949 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.259950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.259950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.260043 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.260092 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.260178 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.260277 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":16}
I20260812 14:27:49.261559 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Bootstrap starting.
I20260812 14:27:49.262399 3384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:49.262642 3384 log.cc:824] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:49.263260 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: No bootstrap required, opened a new log
I20260812 14:27:49.264665 3384 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 32849 } }
I20260812 14:27:49.264787 3384 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:49.264808 3384 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a6c4aee9dd48cc966a5ac49e536b26, State: Initialized, Role: FOLLOWER
I20260812 14:27:49.264904 3384 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 32849 } }
I20260812 14:27:49.265168 3384 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:49.265148 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 32849 } } }
I20260812 14:27:49.265244 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:27:49.266651 3417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:49.266805 3417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:49.268774 3260 master_runner.cc:428] Detected that this master b6a6c4aee9dd48cc966a5ac49e536b26 is joining an existing cluster
I20260812 14:27:49.268810 3260 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.173.252:32849
I20260812 14:27:49.276235 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.269979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.23.173.252:32849)
0812 14:27:49.270016 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.270017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.270371 (+ 354us) server_negotiation.cc:407] Connection header received
0812 14:27:49.270398 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.270399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.270407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.270455 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.270820 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.271522 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272535 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.272798 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272836 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.273888 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.273889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.273889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.273894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.275098 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.275477 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.275477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.275478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.275521 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.275683 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.275683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.275683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.275806 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.275854 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.276010 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.276146 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":21}
I20260812 14:27:49.276247 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.269910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.0.0.1:58710)
0812 14:27:49.270301 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.270310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.270364 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.270499 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.270499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.270676 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.270742 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.270744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.271603 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.271604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.272435 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.272438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.272909 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.272976 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.273357 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.273362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.273789 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.275175 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.275176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.275176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.275405 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.275565 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.275566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.275567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.275629 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.275864 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.275865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.275920 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.275994 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.276102 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":66,"threads_started":1}
I20260812 14:27:49.300870 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 2237
I20260812 14:27:49.304183 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.269766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.23.173.253:38203)
0812 14:27:49.269814 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.269816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.270079 (+ 263us) server_negotiation.cc:407] Connection header received
0812 14:27:49.270191 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.270192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.270202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.270263 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.270630 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.271337 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272631 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.272911 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272947 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.300956 (+ 28009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.300959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.300959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.300968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.302880 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.303310 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.303312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.303312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.303380 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.303573 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.303573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.303574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.303741 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.303787 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.304000 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.304086 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:27:49.304544 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.269703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.0.0.1:59836)
0812 14:27:49.270023 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.270033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.270087 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.270268 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.270269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.270456 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.270544 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.270546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.271425 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.271426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.272524 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.272535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.273020 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.273021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.273080 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.300406 (+ 27326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.300413 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.300831 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.302979 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.302980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.302981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.303221 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.303452 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.303453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.303453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.303508 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.303832 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.303833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.303894 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.304311 (+ 417us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.304438 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":83,"threads_started":1}
I20260812 14:27:49.313519 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.270146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.23.173.254:46677)
0812 14:27:49.270194 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.270196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.270477 (+ 281us) server_negotiation.cc:407] Connection header received
0812 14:27:49.270565 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.270566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.270576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.270606 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.270991 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.271664 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272648 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.272933 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.272967 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.301485 (+ 28518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.301488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.301488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.301498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.312246 (+ 10748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.312657 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.312658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.312658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.312728 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.312924 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.312925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.312926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.313082 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.313159 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.313326 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.313424 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:27:49.313532 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.270163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49432)
0812 14:27:49.270429 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.270437 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.270499 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.270665 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.270666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.270846 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.270906 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.270908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.271799 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.271800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.272565 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.272567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.273041 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.273042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.273095 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.301010 (+ 27915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.301015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.301351 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.312325 (+ 10974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.312326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.312327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.312579 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.312799 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.312799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.312800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.312864 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.313163 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.313165 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.313220 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.313296 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.313413 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":82,"threads_started":1}
I20260812 14:27:49.315765 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:43703
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:46677
--webserver_port=43269
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544868, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.254@KRBTEST.COM for oryx/127.23.173.252@KRBTEST.COM
I20260812 14:27:49.324671 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.316079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.0.0.1:58712)
0812 14:27:49.316163 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.316172 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.316207 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.317379 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.317380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.317548 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.317605 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.317607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.318406 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.318408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.319255 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.319258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.319770 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.319771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.319844 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.320239 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.320244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.321821 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.323570 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.323572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.323572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.323866 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.324080 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.324080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.324081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.324136 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.324384 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.324385 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.324426 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.324485 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.324581 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 14:27:49.324688 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.316277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.23.173.252:32849)
0812 14:27:49.317224 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.317226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.317235 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:49.317270 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.317271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.317281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.317342 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.317687 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.318327 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.319340 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.319662 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.319773 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.321914 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.321916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.321916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.321924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.323448 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.323956 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.323957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.323957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.324016 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.324192 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.324193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.324193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.324322 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.324392 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.324530 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.324602 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922}
I20260812 14:27:49.325263 2121 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.173.252:32849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26"
member_type: NON_VOTER
last_known_addr {
host: "127.23.173.252"
port: 32849
}
attrs {
promote: true
}
}
dest_uuid: "9fc8af0aba90482090c23064205f5fb3"
cas_config_opid_index: 5
I20260812 14:27:49.325528 2121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:49.326017 3096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6a6c4aee9dd48cc966a5ac49e536b26 (127.23.173.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.327031 3096 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.173.252:32849
I20260812 14:27:49.327116 3440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.327204 3440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:49.327638 3260 master.cc:537] Master@127.23.173.252:32849 shutting down...
I20260812 14:27:49.327844 3336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:49.328661 3336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 1 FOLLOWER]: Refusing update from remote peer 9fc8af0aba90482090c23064205f5fb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 14:27:49.332350 3260 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:49.332438 3260 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:49.332481 3260 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: stopping tablet replica
I20260812 14:27:49.332755 3440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:27:49.333629 2087 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 -> Peer b6a6c4aee9dd48cc966a5ac49e536b26 (127.23.173.252:32849): Couldn't send request to peer b6a6c4aee9dd48cc966a5ac49e536b26. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:27:49.336324 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.327255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.0.0.1:59852)
0812 14:27:49.327382 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.327393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.327443 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.331492 (+ 4049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.331493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.331496 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:49.331547 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.331548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.332571 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.332572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.335196 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.335200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.336040 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.336041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.336123 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.336151 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.336208 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.336250 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260812 14:27:49.336315 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.327438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.23.173.253:38203)
0812 14:27:49.331299 (+ 3861us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.331302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.331312 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:49.331348 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.331349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.331374 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.331447 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:49.331645 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.332457 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.335300 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.335957 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.336040 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.336112 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.336212 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.336249 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3831}
I20260812 14:27:49.336578 3191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 LEARNER]: Refusing update from remote peer 9fc8af0aba90482090c23064205f5fb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 14:27:49.336768 3440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 14:27:49.337379 3191 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6a6c4aee9dd48cc966a5ac49e536b26 (127.23.173.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.338657 3231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc8af0aba90482090c23064205f5fb3. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.338752 3231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:27:49.338786 3231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: NON_VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: true } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.338824 3231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:27:49.339412 3440 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: attempting to promote NON_VOTER c1367ca601474caa881012adc5ea6714 to VOTER
I20260812 14:27:49.339600 3440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:49.340131 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.339955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.0.0.1:58724)
0812 14:27:49.340065 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.340103 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:32849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 14:27:49.340427 3191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 LEARNER]: Refusing update from remote peer 9fc8af0aba90482090c23064205f5fb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 14:27:49.340656 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.340524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.0.0.1:58738)
0812 14:27:49.340609 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.340633 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:32849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 14:27:49.340647 3096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260812 14:27:49.340801 2087 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 -> Peer b6a6c4aee9dd48cc966a5ac49e536b26 (127.23.173.252:32849): Couldn't send request to peer b6a6c4aee9dd48cc966a5ac49e536b26. Status: Network error: Client connection negotiation failed: client connection to 127.23.173.252:32849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 14:27:49.341346 3098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c1367ca601474caa881012adc5ea6714 (127.23.173.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.341501 3191 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c1367ca601474caa881012adc5ea6714 (127.23.173.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.342229 3098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.342305 3098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:49.342458 3096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.342518 3096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:49.342733 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fc8af0aba90482090c23064205f5fb3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.342862 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:49.345721 3260 master.cc:552] Master@127.23.173.252:32849 shutdown complete.
I20260812 14:27:49.345775 3260 master_runner.cc:305] Clearing existing system tablet
I20260812 14:27:49.345907 3260 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.345961 3260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.346730 3260 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:49.347265 3447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.347429 3260 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.347491 3260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.347515 3260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
uuid: "b6a6c4aee9dd48cc966a5ac49e536b26"
format_stamp: "Formatted at 2026-08-12 14:27:49 on dist-test-slave-1l3l"
server_key: "bc7b082357ebc6586fc57f088fac4011"
server_key_iv: "9c13a25b0af5953c54276b6edb9275bc"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.348398 3260 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:49.349972 3260 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 14:27:49.350037 3260 log.cc:1196] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 14:27:49.350236 3260 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Deleting consensus metadata
I20260812 14:27:49.350533 3260 master_runner.cc:315] Copying system tablet from 127.23.173.254:46677
I20260812 14:27:49.351074 3260 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: tablet copy: Beginning tablet copy session from remote peer at address 127.23.173.254:46677
I20260812 14:27:49.358716 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.351416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49436)
0812 14:27:49.351533 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.351545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.351618 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.351836 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.351837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.352048 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.352114 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.352117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.353150 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.353151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.354028 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.354032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.354595 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.354596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.354672 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.355184 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.355190 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.355620 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.357577 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.357578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.357579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.357802 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.358033 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.358034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.358034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.358099 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.358385 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.358386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.358440 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.358506 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.358612 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
I20260812 14:27:49.358736 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.351410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.23.173.254:46677)
0812 14:27:49.351459 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.351461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.351622 (+ 161us) server_negotiation.cc:407] Connection header received
0812 14:27:49.351677 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.351678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.351690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.351781 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.352258 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.353042 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.354138 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.354457 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.354505 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.355800 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.355801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.355802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.355809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.357496 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.357888 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.357889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.357890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.357963 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.358163 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.358164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.358164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.358306 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.358392 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.358547 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.358645 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:27:49.358954 2156 tablet_copy_service.cc:140] P 9fc8af0aba90482090c23064205f5fb3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b6a6c4aee9dd48cc966a5ac49e536b26 ({username='oryx', principal='oryx/127.23.173.252@KRBTEST.COM'} at 127.0.0.1:49436)
I20260812 14:27:49.359014 2156 tablet_copy_service.cc:161] P 9fc8af0aba90482090c23064205f5fb3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b6a6c4aee9dd48cc966a5ac49e536b26 at {username='oryx', principal='oryx/127.23.173.252@KRBTEST.COM'} at 127.0.0.1:49436: session id = b6a6c4aee9dd48cc966a5ac49e536b26-00000000000000000000000000000000
I20260812 14:27:49.359601 2156 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:27:49.360107 3260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 14:27:49.361562 3260 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: tablet copy: Starting download of 0 data blocks...
I20260812 14:27:49.361728 3260 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: tablet copy: Starting download of 1 WAL segments...
I20260812 14:27:49.362613 3260 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:27:49.363559 2156 tablet_copy_service.cc:342] P 9fc8af0aba90482090c23064205f5fb3: Request end of tablet copy session b6a6c4aee9dd48cc966a5ac49e536b26-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.173.252@KRBTEST.COM'} at 127.0.0.1:49436
I20260812 14:27:49.363615 2156 tablet_copy_service.cc:433] P 9fc8af0aba90482090c23064205f5fb3: ending tablet copy session b6a6c4aee9dd48cc966a5ac49e536b26-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b6a6c4aee9dd48cc966a5ac49e536b26
I20260812 14:27:49.366119 3260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:49.367426 3461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.368486 3462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.368800 3464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.369025 3260 server_base.cc:1034] running on GCE node
I20260812 14:27:49.369122 3260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:49.369251 3260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:49.370380 3260 hybrid_clock.cc:648] HybridClock initialized: now 1786544869370360 us; error 37 us; skew 500 ppm
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:49.371898 3260 init.cc:377] Logged in from keytab as oryx/127.23.173.252@KRBTEST.COM (short username oryx)
I20260812 14:27:49.372145 3260 webserver.cc:460] Webserver started at http://127.23.173.252:32901/ using document root <none> and password file <none>
I20260812 14:27:49.372251 3260 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.372277 3260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.372975 3260 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:49.373553 3475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.373719 3260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:49.373769 3260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
uuid: "b6a6c4aee9dd48cc966a5ac49e536b26"
format_stamp: "Formatted at 2026-08-12 14:27:49 on dist-test-slave-1l3l"
server_key: "bc7b082357ebc6586fc57f088fac4011"
server_key_iv: "9c13a25b0af5953c54276b6edb9275bc"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.373823 3260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.395807 3260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:49.397709 3260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:49.401113 3260 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.252:32849
I20260812 14:27:49.401216 3578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.252:32849 every 8 connection(s)
I20260812 14:27:49.401474 3260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
I20260812 14:27:49.402427 3579 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:27:49.402980 3579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Bootstrap starting.
I20260812 14:27:49.406018 3579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:27:49.406231 3579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: Bootstrap complete.
I20260812 14:27:49.406414 3579 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:49.406499 3579 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a6c4aee9dd48cc966a5ac49e536b26, State: Initialized, Role: LEARNER
I20260812 14:27:49.406533 3579 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:49.406718 3579 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:49.406711 3582 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } } }
I20260812 14:27:49.406951 3582 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:27:49.408392 3607 catalog_manager.cc:1259] Loaded cluster ID: 9714c2194a8846d4aff3f2ec0a858e06
I20260812 14:27:49.408491 3607 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: loading cluster ID for follower catalog manager: success
I20260812 14:27:49.409898 3607 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: acquiring CA information for follower catalog manager: success
I20260812 14:27:49.410210 3607 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6a6c4aee9dd48cc966a5ac49e536b26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 14:27:49.419405 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:49.419610 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:49.419633 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:49.420836 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:49.420886 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:49.420912 3439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:27:49.422466 3439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:43703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=43269
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:49.422700 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:49.422945 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:49.428013 3612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.428013 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.428056 3614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.428213 3439 server_base.cc:1034] running on GCE node
I20260812 14:27:49.428478 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:49.428695 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:49.429862 3439 hybrid_clock.cc:648] HybridClock initialized: now 1786544869429842 us; error 32 us; skew 500 ppm
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:49.432475 3439 init.cc:377] Logged in from keytab as oryx/127.23.173.193@KRBTEST.COM (short username oryx)
I20260812 14:27:49.432955 3439 webserver.cc:460] Webserver started at http://127.23.173.193:43269/ using document root <none> and password file <none>
I20260812 14:27:49.433166 3439 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.433225 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.434602 3439 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:49.435218 3639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.435389 3439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:49.435467 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "da6868b5df5d451c998fcd4d73180f49"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "5ff7e126efb948e00ed25fc9d0b6f755"
server_key_iv: "70c323f45d0163934649f906c17ec640"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.435871 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.449872 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:49.452186 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:49.452327 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:49.452575 3439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:49.452888 3439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:49.452922 3439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.452952 3439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:49.452976 3439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.458595 3439 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:43703
I20260812 14:27:49.458665 3824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:43703 every 8 connection(s)
I20260812 14:27:49.458973 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:27:49.463358 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 3439
I20260812 14:27:49.463541 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 2462
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:27:49.469614 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.459684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:44331 (local address 127.23.173.254:46677)
0812 14:27:49.459769 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.459771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.460022 (+ 251us) server_negotiation.cc:407] Connection header received
0812 14:27:49.460691 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.460692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.460708 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.460738 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.461636 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.462266 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.463342 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.463612 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.463651 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.466203 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.466204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.466205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.466213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.467913 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.468508 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.468509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.468510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.468578 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.468908 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.468909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.468909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.469045 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.469094 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.469455 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.469532 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 14:27:49.470086 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.459685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:44331)
0812 14:27:49.459905 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.459929 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.460559 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.461149 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.461151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.461350 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.461537 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.461542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.462367 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.462368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.463230 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.463233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.463761 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.463762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.463917 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.464681 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.464691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.466101 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.468016 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.468026 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.468029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.468407 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.468660 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.468661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.468662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.468813 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.469145 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.469153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.469316 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.469727 (+ 411us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.469896 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":25,"spinlock_wait_cycles":9984,"thread_start_us":83,"threads_started":1}
I20260812 14:27:49.470432 3833 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46677
I20260812 14:27:49.470516 3833 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.470678 3833 heartbeater.cc:511] Master 127.23.173.254:46677 requested a full tablet report, sending...
I20260812 14:27:49.471019 2121 ts_manager.cc:194] Re-registered known tserver with Master: da6868b5df5d451c998fcd4d73180f49 (127.23.173.193:43703)
I20260812 14:27:49.471421 2121 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:44331
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.252@KRBTEST.COM
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
I20260812 14:27:49.484126 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.460087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:40267 (local address 127.23.173.252:32849)
0812 14:27:49.460187 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.460189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.460221 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:27:49.460740 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.460742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.460755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.460817 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.462403 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.463430 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.464324 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.466525 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.467054 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.481177 (+ 14123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.481179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.481179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.481187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.482899 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483222 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.483224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.483224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.483282 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483499 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.483501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.483501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.483647 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.483728 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.483923 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.484023 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
I20260812 14:27:49.484143 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:36641
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:46677
--webserver_port=43027
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:27:49.484232 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.459730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:39233 (local address 127.23.173.253:38203)
0812 14:27:49.459770 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.459772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.460022 (+ 250us) server_negotiation.cc:407] Connection header received
0812 14:27:49.460674 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.460675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.460688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.460732 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.461653 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.462145 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.463142 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.463432 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.463495 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.481668 (+ 18173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.481670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.481670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.481679 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.483100 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483415 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.483416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.483417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.483477 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483666 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.483667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.483667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.483820 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.483856 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.484068 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.484150 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:27:49.484473 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.459685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.23.173.193:39233)
0812 14:27:49.459883 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.459929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.460554 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.460817 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.460821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.461303 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.461538 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.461545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.462249 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.462250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.463058 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.463060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.478794 (+ 15734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.478796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.478908 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.479479 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.479485 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.481574 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.483180 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.483181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.483360 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.483544 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.483545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.483599 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.483905 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.483906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.483946 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.484015 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.484114 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":10,"thread_start_us":76,"threads_started":1}
I20260812 14:27:49.484836 3836 heartbeater.cc:347] Connected to a master server at 127.23.173.253:38203
I20260812 14:27:49.484912 3836 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.485010 3836 heartbeater.cc:511] Master 127.23.173.253:38203 requested a full tablet report, sending...
I20260812 14:27:49.485100 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.459993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.23.173.193:40267)
0812 14:27:49.460170 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.460179 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.460562 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.460842 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.460843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.461303 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.461538 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.461545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.463511 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.463512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.464238 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.464241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.466626 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.466627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.468854 (+ 2227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.478923 (+ 10069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.478928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.481065 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.482978 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.482979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.482979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.483139 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.483361 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.483363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.483363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.483437 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.483762 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.483764 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.483813 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.484646 (+ 833us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.484772 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":19,"thread_start_us":63,"threads_started":1}
I20260812 14:27:49.485422 3172 ts_manager.cc:194] Registered new tserver with Master: da6868b5df5d451c998fcd4d73180f49 (127.23.173.193:43703)
I20260812 14:27:49.485438 3831 heartbeater.cc:347] Connected to a master server at 127.23.173.252:32849
I20260812 14:27:49.485482 3831 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.485527 3831 heartbeater.cc:511] Master 127.23.173.252:32849 requested a full tablet report, sending...
I20260812 14:27:49.485922 3518 ts_manager.cc:194] Registered new tserver with Master: da6868b5df5d451c998fcd4d73180f49 (127.23.173.193:43703)
W20260812 14:27:49.566082 3842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:49.566280 3842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:49.566301 3842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:49.567428 3842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:49.567469 3842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:49.567503 3842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:27:49.568980 3842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:36641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=43027
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:49.569193 3842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:49.569413 3842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:49.574345 3852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.574321 3849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.574311 3848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.574326 3842 server_base.cc:1034] running on GCE node
I20260812 14:27:49.574666 3842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:49.574892 3842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:49.576031 3842 hybrid_clock.cc:648] HybridClock initialized: now 1786544869576016 us; error 28 us; skew 500 ppm
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:49.578425 3842 init.cc:377] Logged in from keytab as oryx/127.23.173.194@KRBTEST.COM (short username oryx)
I20260812 14:27:49.578792 3842 webserver.cc:460] Webserver started at http://127.23.173.194:43027/ using document root <none> and password file <none>
I20260812 14:27:49.578964 3842 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.579021 3842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.580399 3842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:49.580911 3867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.581096 3842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.581163 3842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "8002180f25e6495c885ec1a89793c164"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "3bf5c5fe9a0c350ffd9f6818762fbc24"
server_key_iv: "a7606ed76d813bdb586a11cea261ee1b"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.581453 3842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.591846 3842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:49.593636 3842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:49.593750 3842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:49.593943 3842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:49.594239 3842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:49.594274 3842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.594295 3842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:49.594306 3842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.599982 3842 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:36641
I20260812 14:27:49.600039 4069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:36641 every 8 connection(s)
I20260812 14:27:49.600368 3842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.252@KRBTEST.COM
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
I20260812 14:27:49.609781 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 3842
I20260812 14:27:49.609943 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 2680
I20260812 14:27:49.628321 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.601496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:38605 (local address 127.23.173.253:38203)
0812 14:27:49.601554 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.601556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.601713 (+ 157us) server_negotiation.cc:407] Connection header received
0812 14:27:49.602021 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.602023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.602034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.602066 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.603045 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.603533 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.604874 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.605102 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.605144 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.608604 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.608605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.608606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.608614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.610103 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.627254 (+ 17151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.627256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.627256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.627324 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.627606 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.627607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.627607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.627762 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.627808 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.628149 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.628237 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:27:49.628877 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.601401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.23.173.194:38605)
0812 14:27:49.601660 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.601666 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.601899 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.602162 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.602163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.602708 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.602922 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.602928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.603652 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.603653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.604768 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.604771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.605218 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.605219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.605273 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.605858 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.605863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.608517 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.626859 (+ 18342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.626861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.626862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.627129 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.627385 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.627385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.627386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.627545 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.627838 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.627846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.628021 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.628414 (+ 393us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.628628 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":7,"thread_start_us":56,"threads_started":1}
I20260812 14:27:49.631143 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.601164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:51121 (local address 127.23.173.252:32849)
0812 14:27:49.601210 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.601212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.601393 (+ 181us) server_negotiation.cc:407] Connection header received
0812 14:27:49.602019 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.602021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.602037 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.602066 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.603048 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.603844 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.604701 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.604946 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.605020 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.608634 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.608635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.608636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.608643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.610236 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.630032 (+ 19796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.630035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.630035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.630125 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.630399 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.630400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.630401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.630571 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.630628 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.630896 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.631037 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:27:49.631907 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.601085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.23.173.194:51121)
0812 14:27:49.601275 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.601313 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.601889 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.602157 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.602162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.602699 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.602922 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.602928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.603927 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.603928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.604612 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.604614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.605032 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.605033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.605182 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.605931 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.605935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.608528 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.628878 (+ 20350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.628879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.628879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.629880 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.630223 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.630224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.630224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.630301 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.630687 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.630688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.630744 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.631625 (+ 881us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.631779 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":51,"thread_start_us":94,"threads_started":1}
I20260812 14:27:49.633998 4071 heartbeater.cc:347] Connected to a master server at 127.23.173.252:32849
I20260812 14:27:49.634097 4071 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.634272 4071 heartbeater.cc:511] Master 127.23.173.252:32849 requested a full tablet report, sending...
I20260812 14:27:49.634631 3518 ts_manager.cc:194] Registered new tserver with Master: 8002180f25e6495c885ec1a89793c164 (127.23.173.194:36641)
I20260812 14:27:49.635082 4076 heartbeater.cc:347] Connected to a master server at 127.23.173.253:38203
I20260812 14:27:49.635143 4076 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.635206 4076 heartbeater.cc:511] Master 127.23.173.253:38203 requested a full tablet report, sending...
I20260812 14:27:49.635535 3172 ts_manager.cc:194] Registered new tserver with Master: 8002180f25e6495c885ec1a89793c164 (127.23.173.194:36641)
I20260812 14:27:49.636081 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.601104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:38609 (local address 127.23.173.254:46677)
0812 14:27:49.601176 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.601178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.601618 (+ 440us) server_negotiation.cc:407] Connection header received
0812 14:27:49.601997 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.601999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.602013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.602066 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.603024 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.603818 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.604859 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.605117 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.605171 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.607643 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.607644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.607644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.607651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.609238 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.635106 (+ 25868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.635108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.635108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.635193 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.635457 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.635459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.635459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.635608 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.635784 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.635921 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.636017 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:27:49.636091 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.601086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:38609)
0812 14:27:49.601546 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.601556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.601891 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.602432 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.602433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.602683 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.602921 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.602926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.603900 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.603901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.604786 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.604788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.605185 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.605186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.605252 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.605828 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.605838 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.607512 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.609669 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.609676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.609679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.609967 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.635287 (+ 25320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.635288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.635288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.635377 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.635770 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.635771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.635818 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.635884 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.636002 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":92,"thread_start_us":70,"threads_started":1}
I20260812 14:27:49.636623 4072 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46677
I20260812 14:27:49.636677 4072 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.636734 4072 heartbeater.cc:511] Master 127.23.173.254:46677 requested a full tablet report, sending...
I20260812 14:27:49.636971 2121 ts_manager.cc:194] Re-registered known tserver with Master: 8002180f25e6495c885ec1a89793c164 (127.23.173.194:36641)
I20260812 14:27:49.637332 2121 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:38609
I20260812 14:27:49.639520 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:35449
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:46677
--webserver_port=34453
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:49.721968 4086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:49.722194 4086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:49.722219 4086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:49.723354 4086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:49.723392 4086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:49.723416 4086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:27:49.724917 4086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:35449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=34453
--tserver_master_addrs=127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:49.725131 4086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:49.725317 4086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:49.730161 4101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.730162 4094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:49.730190 4095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:49.730284 4086 server_base.cc:1034] running on GCE node
I20260812 14:27:49.730613 4086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:49.730845 4086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:49.731979 4086 hybrid_clock.cc:648] HybridClock initialized: now 1786544869731963 us; error 33 us; skew 500 ppm
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:49.734449 4086 init.cc:377] Logged in from keytab as oryx/127.23.173.195@KRBTEST.COM (short username oryx)
I20260812 14:27:49.734872 4086 webserver.cc:460] Webserver started at http://127.23.173.195:34453/ using document root <none> and password file <none>
I20260812 14:27:49.735078 4086 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:49.735146 4086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:49.736617 4086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:49.737160 4111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.737305 4086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:49.737385 4086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0"
format_stamp: "Formatted at 2026-08-12 14:27:48 on dist-test-slave-1l3l"
server_key: "aeb496dd356118a6213b9c43b78ddea1"
server_key_iv: "9fb814e8dfc6b20c06bce1b23ca0fca3"
server_key_version: "encryptionkey@0"
I20260812 14:27:49.737716 4086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:49.760535 4086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:49.762480 4086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:49.762634 4086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:49.762859 4086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:27:49.763165 4086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:27:49.763207 4086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.763237 4086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:27:49.763257 4086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:49.768915 4086 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:35449
I20260812 14:27:49.768975 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:35449 every 8 connection(s)
I20260812 14:27:49.769309 4086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:27:49.774675 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 4086
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.253@KRBTEST.COM
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.252@KRBTEST.COM
Aug 12 14:27:49 dist-test-slave-1l3l krb5kdc[2036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544869, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:27:49.782641 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.770063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:32849 (local address 127.23.173.195:42699)
0812 14:27:49.770245 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.770283 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.771327 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.771631 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.771635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.772055 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.772272 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.772277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.773275 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.773276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.774558 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.774560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.775086 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.775087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.775231 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.776290 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.776295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.779156 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.780937 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.780944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.780947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.781247 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.781488 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.781497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.781497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.781633 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.781929 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.781937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.782105 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.782232 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.782397 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260812 14:27:49.782953 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.770038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:42699 (local address 127.23.173.252:32849)
0812 14:27:49.770325 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.770327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.770368 (+ 41us) server_negotiation.cc:407] Connection header received
0812 14:27:49.771434 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.771436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.771450 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.771498 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.772371 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.773177 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.774646 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.774959 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.775001 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.779232 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.779234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.779234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.779242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.780863 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.781359 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.781361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.781361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.781429 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.781712 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.781713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.781714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.781842 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.781901 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.782787 (+ 886us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.782868 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260812 14:27:49.783313 4302 heartbeater.cc:347] Connected to a master server at 127.23.173.252:32849
I20260812 14:27:49.783416 4302 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.783618 4302 heartbeater.cc:511] Master 127.23.173.252:32849 requested a full tablet report, sending...
I20260812 14:27:49.784072 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.770359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:38203 (local address 127.23.173.195:43045)
0812 14:27:49.770495 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.770505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.771345 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.771633 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.771636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.772069 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.772273 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.772279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.773275 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.773276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.774538 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.774541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.775183 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.775183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.775236 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.776053 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.776058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.779413 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.783002 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.783003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.783003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.783168 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.783350 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.783351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.783351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.783423 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.783696 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.783697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.783771 (+ 74us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.783836 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.783943 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":57,"thread_start_us":75,"threads_started":1}
I20260812 14:27:49.784168 3518 ts_manager.cc:194] Registered new tserver with Master: 5ea4495ded204ab3a688d5d1cd9b0ce0 (127.23.173.195:35449)
I20260812 14:27:49.784612 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.770553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:43045 (local address 127.23.173.253:38203)
0812 14:27:49.770623 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.770625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.770634 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:49.771451 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.771452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.771466 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.771497 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.772414 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.773166 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.774646 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.774964 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.775001 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.779484 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.779486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.779486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.779500 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.782924 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.783235 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.783236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.783236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.783290 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.783496 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.783497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.783508 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.783628 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.783681 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.784491 (+ 810us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.784554 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:27:49.784988 4304 heartbeater.cc:347] Connected to a master server at 127.23.173.253:38203
I20260812 14:27:49.785068 4304 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.785099 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.770062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:32911)
0812 14:27:49.770238 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.770283 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.771333 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.771634 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.771636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.772061 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:49.772273 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.772279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.773808 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.773809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.774554 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.774558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.775077 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.775077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.775231 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.776047 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:49.776057 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:49.778999 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:49.783579 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.783580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.783580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.783793 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.784346 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:49.784347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:49.784348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:49.784409 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:49.784685 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:49.784686 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:49.784728 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.784783 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.784883 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260812 14:27:49.785137 4304 heartbeater.cc:511] Master 127.23.173.253:38203 requested a full tablet report, sending...
I20260812 14:27:49.785228 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.770053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:32911 (local address 127.23.173.254:46677)
0812 14:27:49.770106 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.770108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.770365 (+ 257us) server_negotiation.cc:407] Connection header received
0812 14:27:49.771503 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.771505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.771520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.771576 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.773262 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.773729 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.774659 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.774939 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.774983 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.779099 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.779100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.779101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.779109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.783507 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.784167 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.784168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.784169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.784254 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.784502 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.784504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.784504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.784620 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.784681 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.785064 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.785143 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:27:49.785501 3172 ts_manager.cc:194] Registered new tserver with Master: 5ea4495ded204ab3a688d5d1cd9b0ce0 (127.23.173.195:35449)
I20260812 14:27:49.785794 4303 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46677
I20260812 14:27:49.785851 4303 heartbeater.cc:464] Registering TS with master...
I20260812 14:27:49.785923 4303 heartbeater.cc:511] Master 127.23.173.254:46677 requested a full tablet report, sending...
I20260812 14:27:49.786161 2121 ts_manager.cc:194] Re-registered known tserver with Master: 5ea4495ded204ab3a688d5d1cd9b0ce0 (127.23.173.195:35449)
I20260812 14:27:49.786633 2121 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:32911
I20260812 14:27:49.787971 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.776160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.23.173.252:32849)
0812 14:27:49.776274 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.776275 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.777425 (+ 1150us) server_negotiation.cc:407] Connection header received
0812 14:27:49.777522 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.777523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.777536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.777572 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.778229 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.779828 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.780707 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.781001 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.781042 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.782419 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.782420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.782421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.782427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.786681 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.787053 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.787054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.787054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.787116 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.787331 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.787332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.787332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.787468 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.787559 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.787801 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.787884 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260812 14:27:49.789754 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.775883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.23.173.253:38203)
0812 14:27:49.777225 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.777226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.777232 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:27:49.777247 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.777247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.777256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.777305 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.777916 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.778640 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.781113 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.781403 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.781446 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.786904 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.786905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.786906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.786912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.788718 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.789034 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.789035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.789036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.789115 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.789261 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.789262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.789262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.789387 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.789484 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.789584 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.789665 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":63,"threads_started":1}
I20260812 14:27:49.790457 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.775742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.23.173.254:46677)
0812 14:27:49.777064 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.777066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.777074 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:49.777106 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.777106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.777118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.777166 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.777854 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.778634 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.781763 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.782061 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.782102 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.787642 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.787644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.787644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.787652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.789154 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.789639 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.789640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.789640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.789745 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.789994 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.789995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.789995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.790127 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.790215 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.790312 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.790379 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1295,"thread_start_us":96,"threads_started":1}
I20260812 14:27:49.791764 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:27:49.793305 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:27:49.800822 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.797280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.23.173.194:36641)
0812 14:27:49.797417 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.797421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.797428 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:49.797460 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.797462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.797479 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.797689 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:49.797745 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.798735 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.799772 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.800523 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.800585 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.800645 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.800706 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.800745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260812 14:27:49.800823 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.796960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:36641 (local address 127.0.0.1:46190)
0812 14:27:49.797026 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.797037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.797073 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.797581 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.797581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.797583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:49.797629 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.797631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.798820 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.798821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.799645 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.799647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.800579 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.800580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.800634 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.800637 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.800690 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.800729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260812 14:27:49.800895 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.796961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35449 (local address 127.0.0.1:55366)
0812 14:27:49.797179 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.797188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.797212 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.797744 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.797745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.797746 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:49.797775 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.797776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.799046 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.799048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.799977 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.799979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.800640 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.800641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.800699 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.800703 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.800731 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.800768 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":19}
I20260812 14:27:49.801294 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.797430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.23.173.195:35449)
0812 14:27:49.797584 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.797589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.797600 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:49.797638 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.797640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.797656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.797741 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:49.797833 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.798934 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.800060 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.800549 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.801060 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.801130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.801173 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.801213 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260812 14:27:49.801681 3950 tablet_service.cc:1467] Processing CreateTablet for tablet 02bc6e9894bd421e9eaeaae01cfa264c (DEFAULT_TABLE table=test-table [id=e8fbbe1fb90b40fcabe0d176fad3d2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:49.801838 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.797055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43703 (local address 127.0.0.1:34704)
0812 14:27:49.797191 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.797201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.797243 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.798100 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.798101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.798103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:49.798146 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.798147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.799339 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.799340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.800424 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.800427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.801617 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.801618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.801678 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.801682 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.801734 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.801773 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":4,"thread_start_us":92,"threads_started":1}
I20260812 14:27:49.801854 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.797688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.23.173.193:43703)
0812 14:27:49.797936 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.797940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.797949 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:49.797985 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.797987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.798003 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.798211 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:49.798265 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.799262 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.800524 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.801550 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.801634 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.801685 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.801748 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.801783 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260812 14:27:49.801916 4193 tablet_service.cc:1467] Processing CreateTablet for tablet 02bc6e9894bd421e9eaeaae01cfa264c (DEFAULT_TABLE table=test-table [id=e8fbbe1fb90b40fcabe0d176fad3d2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:49.802080 3950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bc6e9894bd421e9eaeaae01cfa264c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:49.802219 4193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bc6e9894bd421e9eaeaae01cfa264c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:49.802541 3718 tablet_service.cc:1467] Processing CreateTablet for tablet 02bc6e9894bd421e9eaeaae01cfa264c (DEFAULT_TABLE table=test-table [id=e8fbbe1fb90b40fcabe0d176fad3d2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:27:49.802791 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bc6e9894bd421e9eaeaae01cfa264c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:49.805404 4347 tablet_bootstrap.cc:492] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Bootstrap starting.
I20260812 14:27:49.805879 4348 tablet_bootstrap.cc:492] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Bootstrap starting.
I20260812 14:27:49.806083 4347 tablet_bootstrap.cc:654] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:49.806430 4347 log.cc:824] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:49.806535 4348 tablet_bootstrap.cc:654] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:49.806866 4348 log.cc:824] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:49.807194 4350 tablet_bootstrap.cc:492] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Bootstrap starting.
I20260812 14:27:49.807294 4347 tablet_bootstrap.cc:492] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: No bootstrap required, opened a new log
I20260812 14:27:49.807358 4347 ts_tablet_manager.cc:1397] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:49.807693 4348 tablet_bootstrap.cc:492] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: No bootstrap required, opened a new log
I20260812 14:27:49.807780 4348 ts_tablet_manager.cc:1397] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:27:49.807884 4350 tablet_bootstrap.cc:654] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:49.808188 4350 log.cc:824] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:49.809621 4348 raft_consensus.cc:348] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.809660 4347 raft_consensus.cc:348] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.809753 4348 raft_consensus.cc:374] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:49.809770 4347 raft_consensus.cc:374] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:49.809787 4348 raft_consensus.cc:729] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea4495ded204ab3a688d5d1cd9b0ce0, State: Initialized, Role: FOLLOWER
I20260812 14:27:49.809790 4347 raft_consensus.cc:729] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6868b5df5d451c998fcd4d73180f49, State: Initialized, Role: FOLLOWER
I20260812 14:27:49.809902 4347 consensus_queue.cc:260] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.809899 4348 consensus_queue.cc:260] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.810154 4347 ts_tablet_manager.cc:1428] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:27:49.810177 4348 ts_tablet_manager.cc:1428] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:27:49.810243 3833 heartbeater.cc:503] Master 127.23.173.254:46677 was elected leader, sending a full tablet report...
I20260812 14:27:49.810389 4303 heartbeater.cc:503] Master 127.23.173.254:46677 was elected leader, sending a full tablet report...
I20260812 14:27:49.812069 4350 tablet_bootstrap.cc:492] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: No bootstrap required, opened a new log
I20260812 14:27:49.812134 4350 ts_tablet_manager.cc:1397] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 14:27:49.813596 4350 raft_consensus.cc:348] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.813717 4350 raft_consensus.cc:374] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:49.813748 4350 raft_consensus.cc:729] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8002180f25e6495c885ec1a89793c164, State: Initialized, Role: FOLLOWER
I20260812 14:27:49.813855 4350 consensus_queue.cc:260] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.814090 4350 ts_tablet_manager.cc:1428] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:27:49.814132 4072 heartbeater.cc:503] Master 127.23.173.254:46677 was elected leader, sending a full tablet report...
W20260812 14:27:49.851073 4077 tablet.cc:2369] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:27:49.858392 4361 raft_consensus.cc:482] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:27:49.858525 4361 raft_consensus.cc:504] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.858897 4361 leader_election.cc:288] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea4495ded204ab3a688d5d1cd9b0ce0 (127.23.173.195:35449), da6868b5df5d451c998fcd4d73180f49 (127.23.173.193:43703)
I20260812 14:27:49.861630 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.859133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:34855 (local address 127.23.173.195:35449)
0812 14:27:49.859179 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.859181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.859190 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:49.859249 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.859250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.859259 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.859303 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:49.859459 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.860009 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.860951 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.861309 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.861366 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.861375 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.861524 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.861562 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:27:49.861644 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.859162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:52159 (local address 127.23.173.193:43703)
0812 14:27:49.859214 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.859216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.859247 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:27:49.859272 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.859273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.859280 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.859325 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:27:49.859467 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.859997 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.860952 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.861328 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.861392 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.861401 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.861542 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.861580 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:27:49.861660 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.859103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43703 (local address 127.23.173.194:52159)
0812 14:27:49.859158 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.859169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.859210 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.859335 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.859336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.859337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:49.859403 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.859404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.860077 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.860078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.860854 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.860856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.861408 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.861409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.861470 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.861474 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.861523 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.861563 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":40}
I20260812 14:27:49.861662 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.859068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35449 (local address 127.23.173.194:34855)
0812 14:27:49.859134 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:49.859146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:49.859190 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:49.859317 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:49.859318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:49.859319 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:27:49.859394 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.859396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.860084 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.860085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.860853 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:49.860856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.861379 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.861381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:49.861455 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.861458 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:27:49.861508 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:27:49.861552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 14:27:49.862025 3748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bc6e9894bd421e9eaeaae01cfa264c" candidate_uuid: "8002180f25e6495c885ec1a89793c164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6868b5df5d451c998fcd4d73180f49" is_pre_election: true
I20260812 14:27:49.862126 4218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bc6e9894bd421e9eaeaae01cfa264c" candidate_uuid: "8002180f25e6495c885ec1a89793c164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" is_pre_election: true
I20260812 14:27:49.862206 3748 raft_consensus.cc:2454] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8002180f25e6495c885ec1a89793c164 in term 0.
I20260812 14:27:49.862251 4218 raft_consensus.cc:2454] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8002180f25e6495c885ec1a89793c164 in term 0.
I20260812 14:27:49.862478 3870 leader_election.cc:302] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea4495ded204ab3a688d5d1cd9b0ce0, 8002180f25e6495c885ec1a89793c164; no voters:
I20260812 14:27:49.862624 4361 raft_consensus.cc:2781] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:27:49.862684 4361 raft_consensus.cc:482] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:27:49.862708 4361 raft_consensus.cc:3037] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:49.863408 4361 raft_consensus.cc:504] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.863538 4361 leader_election.cc:288] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea4495ded204ab3a688d5d1cd9b0ce0 (127.23.173.195:35449), da6868b5df5d451c998fcd4d73180f49 (127.23.173.193:43703)
I20260812 14:27:49.863718 4218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bc6e9894bd421e9eaeaae01cfa264c" candidate_uuid: "8002180f25e6495c885ec1a89793c164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0"
I20260812 14:27:49.863751 3748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bc6e9894bd421e9eaeaae01cfa264c" candidate_uuid: "8002180f25e6495c885ec1a89793c164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6868b5df5d451c998fcd4d73180f49"
I20260812 14:27:49.863812 3748 raft_consensus.cc:3037] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:49.863806 4218 raft_consensus.cc:3037] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:49.864679 4218 raft_consensus.cc:2454] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8002180f25e6495c885ec1a89793c164 in term 1.
I20260812 14:27:49.864682 3748 raft_consensus.cc:2454] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8002180f25e6495c885ec1a89793c164 in term 1.
I20260812 14:27:49.864854 3873 leader_election.cc:302] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8002180f25e6495c885ec1a89793c164, da6868b5df5d451c998fcd4d73180f49; no voters:
I20260812 14:27:49.864948 4361 raft_consensus.cc:2781] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:49.865087 4361 raft_consensus.cc:686] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 1 LEADER]: Becoming Leader. State: Replica: 8002180f25e6495c885ec1a89793c164, State: Running, Role: LEADER
I20260812 14:27:49.865172 4361 consensus_queue.cc:237] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } }
I20260812 14:27:49.866042 2121 catalog_manager.cc:5495] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8002180f25e6495c885ec1a89793c164 (127.23.173.194). New cstate: current_term: 1 leader_uuid: "8002180f25e6495c885ec1a89793c164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002180f25e6495c885ec1a89793c164" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 36641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 } health_report { overall_health: UNKNOWN } } }
I20260812 14:27:49.872701 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.870664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.23.173.194:36641)
0812 14:27:49.870708 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.870709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.870717 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:49.870780 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.870780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.870795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.870849 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:49.870949 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.871399 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.872111 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.872286 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.872332 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.872437 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:49.872537 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:49.872598 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.872616 (+ 18us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.872637 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:27:49.874358 4218 raft_consensus.cc:1261] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 1 FOLLOWER]: Refusing update from remote peer 8002180f25e6495c885ec1a89793c164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:49.874354 3748 raft_consensus.cc:1261] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 1 FOLLOWER]: Refusing update from remote peer 8002180f25e6495c885ec1a89793c164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:49.874692 4361 consensus_queue.cc:1035] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea4495ded204ab3a688d5d1cd9b0ce0" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:49.874840 4363 consensus_queue.cc:1035] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6868b5df5d451c998fcd4d73180f49" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:49.877324 4366 mvcc.cc:205] Tried to move back new op lower bound from 7317687787002978304 to 7317687786968281088. Current Snapshot: MvccSnapshot[applied={T|T < 7317687787002978304}]
I20260812 14:27:49.880302 4368 mvcc.cc:205] Tried to move back new op lower bound from 7317687787002978304 to 7317687786968281088. Current Snapshot: MvccSnapshot[applied={T|T < 7317687787002978304}]
I20260812 14:27:49.880862 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:49.880957 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:27:49.882125 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fc8af0aba90482090c23064205f5fb3: Sending DeleteTablet for 3 replicas of tablet 02bc6e9894bd421e9eaeaae01cfa264c
I20260812 14:27:49.882493 3718 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bc6e9894bd421e9eaeaae01cfa264c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:49 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:34704
I20260812 14:27:49.882490 3950 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bc6e9894bd421e9eaeaae01cfa264c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:49 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:46190
I20260812 14:27:49.882731 4380 tablet_replica.cc:330] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: stopping tablet replica
I20260812 14:27:49.882841 4380 raft_consensus.cc:2229] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:49.882875 4193 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bc6e9894bd421e9eaeaae01cfa264c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:27:49 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:55366
I20260812 14:27:49.882947 4380 raft_consensus.cc:2258] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:49.883109 4383 tablet_replica.cc:330] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: stopping tablet replica
I20260812 14:27:49.883121 4378 tablet_replica.cc:330] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: stopping tablet replica
I20260812 14:27:49.883203 4378 raft_consensus.cc:2229] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:27:49.883298 4378 raft_consensus.cc:2258] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:49.883302 4383 raft_consensus.cc:2229] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:27:49.883460 4383 raft_consensus.cc:2258] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:27:49.883986 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 2062
I20260812 14:27:49.895560 4378 ts_tablet_manager.cc:1905] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:49.897555 4378 ts_tablet_manager.cc:1918] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:49.897632 4378 log.cc:1196] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/02bc6e9894bd421e9eaeaae01cfa264c
I20260812 14:27:49.897941 4378 ts_tablet_manager.cc:1939] T 02bc6e9894bd421e9eaeaae01cfa264c P 8002180f25e6495c885ec1a89793c164: Deleting consensus metadata
I20260812 14:27:49.900229 4380 ts_tablet_manager.cc:1905] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:27:49.901646 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.883815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.23.173.252:32849)
0812 14:27:49.883856 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.883857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.884102 (+ 245us) server_negotiation.cc:407] Connection header received
0812 14:27:49.894345 (+ 10243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.894347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.894362 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.894388 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.895064 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.895792 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.896772 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.897054 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.897103 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.898752 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.898753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.898753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.898760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.900237 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.900744 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.900745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.900746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.900802 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.901017 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.901018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.901018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.901155 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.901235 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.901463 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.901551 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:49.901926 4380 ts_tablet_manager.cc:1918] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:49.901998 4380 log.cc:1196] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/02bc6e9894bd421e9eaeaae01cfa264c
I20260812 14:27:49.902254 4380 ts_tablet_manager.cc:1939] T 02bc6e9894bd421e9eaeaae01cfa264c P da6868b5df5d451c998fcd4d73180f49: Deleting consensus metadata
W20260812 14:27:49.902554 3868 connection.cc:535] server connection from 127.0.0.1:46190 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:27:49.903240 4383 ts_tablet_manager.cc:1905] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:27:49.903272 3642 connection.cc:441] server connection from 127.0.0.1:34704 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34704 (request call id 1) could send its response
I20260812 14:27:49.905045 4383 ts_tablet_manager.cc:1918] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:27:49.905246 4383 log.cc:1196] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/02bc6e9894bd421e9eaeaae01cfa264c
I20260812 14:27:49.905572 4383 ts_tablet_manager.cc:1939] T 02bc6e9894bd421e9eaeaae01cfa264c P 5ea4495ded204ab3a688d5d1cd9b0ce0: Deleting consensus metadata
W20260812 14:27:49.905949 4117 connection.cc:441] server connection from 127.0.0.1:55366 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55366 (request call id 1) could send its response
I20260812 14:27:49.911540 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.903697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.23.173.253:38203)
0812 14:27:49.903770 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.903771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.904084 (+ 313us) server_negotiation.cc:407] Connection header received
0812 14:27:49.904179 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.904180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.904194 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.904222 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.905342 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.906023 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.906947 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.907249 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.907309 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.908938 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.908939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.908940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.908956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.910471 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.910783 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.910785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.910785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.910842 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.911010 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.911011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.911012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.911134 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.911228 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.911340 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.911434 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:27:49.912029 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:49.903945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.23.173.252:32849)
0812 14:27:49.903979 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:49.903980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:49.904085 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:27:49.904176 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:49.904176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:49.904186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:49.904213 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:49.905010 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.905725 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.906838 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:49.907202 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:49.907273 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:49.909244 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:49.909245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:49.909245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:49.909253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:49.910743 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.911137 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.911138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.911139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.911190 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:49.911384 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:49.911385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:49.911386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:49.911502 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:49.911563 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:49.911863 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 14:27:49.911938 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:27:50.878816 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:50.878545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:46073)
0812 14:27:50.878669 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:50.878761 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260812 14:27:50.879025 4303 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:46677 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:50.881276 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:50.881030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:37315)
0812 14:27:50.881149 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:50.881231 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260812 14:27:50.881291 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:50.880818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:46279)
0812 14:27:50.881160 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:50.881240 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":52,"threads_started":1}
W20260812 14:27:50.881469 4072 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:46677 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
W20260812 14:27:50.881467 3833 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:46677 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:51.383450 4406 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:27:51.383603 4406 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:51.383904 4406 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:27:51.384337 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:51.384065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49916)
0812 14:27:51.384219 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:51.384295 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
W20260812 14:27:51.384575 3152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:51.385058 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:51.384972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:49924)
0812 14:27:51.385006 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:51.385035 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260812 14:27:51.385145 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:51.385205 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:27:51.385301 4406 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:27:51.879809 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:51.879487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:33419)
0812 14:27:51.879678 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:51.879745 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
I20260812 14:27:51.882100 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:51.881856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:34101)
0812 14:27:51.882019 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:51.882053 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":43,"threads_started":1}
I20260812 14:27:51.882301 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:51.881814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:36181)
0812 14:27:51.882086 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:51.882126 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":84,"threads_started":1}
I20260812 14:27:52.880419 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:52.880207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:38131)
0812 14:27:52.880327 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:52.880371 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260812 14:27:52.882767 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:52.882492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:51011)
0812 14:27:52.882675 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:52.882719 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260812 14:27:52.882938 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:52.882739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:54105)
0812 14:27:52.882856 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:52.882901 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260812 14:27:53.270627 4421 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:27:53.270740 4421 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:53.270902 4421 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:27:53.271273 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:53.271090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:50160)
0812 14:27:53.271194 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:53.271233 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
I20260812 14:27:53.271744 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:53.271623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:50164)
0812 14:27:53.271670 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:53.271698 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:27:53.271834 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:53.271888 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:27:53.271970 4421 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:27:53.881053 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:53.880820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:48545)
0812 14:27:53.880944 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:53.880989 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260812 14:27:53.883379 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:53.883162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:52373)
0812 14:27:53.883284 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:53.883330 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20260812 14:27:53.883469 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:53.883276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:33457)
0812 14:27:53.883389 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:53.883424 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":97,"threads_started":1}
I20260812 14:27:54.881685 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:54.881438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:48973)
0812 14:27:54.881589 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:54.881636 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260812 14:27:54.884016 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:54.883831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:46099)
0812 14:27:54.883928 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:54.883969 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20260812 14:27:54.884016 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:54.883802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:50723)
0812 14:27:54.883930 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:54.883967 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260812 14:27:55.882368 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:55.882078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:58595)
0812 14:27:55.882251 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:55.882299 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260812 14:27:55.884580 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:55.884350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:53549)
0812 14:27:55.884467 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:55.884529 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260812 14:27:55.884614 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:55.884405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:44509)
0812 14:27:55.884537 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:55.884576 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":45,"threads_started":1}
I20260812 14:27:56.304046 4444 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:27:56.304329 4444 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:56.304513 4444 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:27:56.304912 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:56.304707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:50444)
0812 14:27:56.304833 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:56.304875 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260812 14:27:56.305317 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:56.305226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:50450)
0812 14:27:56.305269 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:56.305298 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:27:56.305384 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:56.305434 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:27:56.305531 4444 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:27:56.883060 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:56.882775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:40749)
0812 14:27:56.882940 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:56.882993 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260812 14:27:56.885243 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:56.884998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:40337)
0812 14:27:56.885165 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:56.885204 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
I20260812 14:27:56.885247 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:56.885039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:52121)
0812 14:27:56.885170 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:56.885208 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260812 14:27:57.883808 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:57.883524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:52183)
0812 14:27:57.883660 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:57.883744 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260812 14:27:57.885891 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:57.885680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:55877)
0812 14:27:57.885807 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:57.885849 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260812 14:27:57.885891 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:57.885680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:60643)
0812 14:27:57.885814 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:57.885850 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260812 14:27:58.884572 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:58.884285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:40901)
0812 14:27:58.884441 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:58.884494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260812 14:27:58.886600 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:58.886350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:36257)
0812 14:27:58.886493 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:58.886557 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260812 14:27:58.886687 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:58.886511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:43563)
0812 14:27:58.886615 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:58.886646 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":43,"threads_started":1}
I20260812 14:27:59.491854 4466 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:27:59.491974 4466 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:27:59.492130 4466 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:27:59.492473 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:59.492275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:33434)
0812 14:27:59.492394 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:59.492432 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
W20260812 14:27:59.492574 3152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260812 14:27:59.492933 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:59.492825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:33436)
0812 14:27:59.492884 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:59.492910 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 14:27:59.493002 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:27:59.493036 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:27:59.493101 4466 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:27:59.885277 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:59.885000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:40171)
0812 14:27:59.885150 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:59.885205 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260812 14:27:59.887233 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:59.887030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:39461)
0812 14:27:59.887157 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:59.887193 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260812 14:27:59.887239 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:59.887044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:49187)
0812 14:27:59.887168 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:59.887202 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260812 14:28:00.885996 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:00.885708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:56911)
0812 14:28:00.885856 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:00.885909 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260812 14:28:00.887791 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:00.887601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:42557)
0812 14:28:00.887692 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:00.887752 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
I20260812 14:28:00.887799 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:00.887582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:51421)
0812 14:28:00.887696 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:00.887757 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260812 14:28:01.886744 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:01.886472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:59321)
0812 14:28:01.886628 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:01.886680 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260812 14:28:01.888352 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:01.888155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:59039)
0812 14:28:01.888269 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:01.888311 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":50,"threads_started":1}
I20260812 14:28:01.888353 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:01.888180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:52137)
0812 14:28:01.888281 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:01.888314 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
I20260812 14:28:02.887570 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:02.887314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:49973)
0812 14:28:02.887445 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:02.887508 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260812 14:28:02.888917 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:02.888746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:43241)
0812 14:28:02.888846 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:02.888882 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260812 14:28:02.888918 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:02.888726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:35301)
0812 14:28:02.888845 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:02.888879 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260812 14:28:03.888283 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:03.888019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:48387)
0812 14:28:03.888190 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:03.888232 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260812 14:28:03.889571 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:03.889371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:52869)
0812 14:28:03.889491 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:03.889523 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260812 14:28:03.889571 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:03.889360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:55569)
0812 14:28:03.889486 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:03.889532 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260812 14:28:04.618465 4491 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:28:04.618595 4491 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:28:04.618742 4491 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:28:04.619165 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:04.618954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:33748)
0812 14:28:04.619084 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:04.619124 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
W20260812 14:28:04.619254 3152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:04.619673 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:04.619557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:33756)
0812 14:28:04.619616 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:04.619648 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 14:28:04.619778 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:04.619822 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:28:04.619905 4491 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:28:04.889371 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:04.888943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:47805)
0812 14:28:04.889239 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:04.889297 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260812 14:28:04.890157 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:04.889980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:37057)
0812 14:28:04.890092 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:04.890121 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260812 14:28:04.890188 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:04.889982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:35583)
0812 14:28:04.890119 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:04.890153 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260812 14:28:05.890303 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:05.889979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:38721)
0812 14:28:05.890196 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:05.890242 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
I20260812 14:28:05.890774 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:05.890540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:40673)
0812 14:28:05.890687 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:05.890726 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":52,"threads_started":1}
I20260812 14:28:05.890853 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:05.890661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:38049)
0812 14:28:05.890776 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:05.890809 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260812 14:28:06.891263 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:06.890862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:58847)
0812 14:28:06.891074 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:06.891187 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260812 14:28:06.891530 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:06.891287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:44523)
0812 14:28:06.891446 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:06.891482 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":41,"threads_started":1}
I20260812 14:28:06.891590 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:06.891254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:53919)
0812 14:28:06.891508 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:06.891547 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":57,"threads_started":1}
I20260812 14:28:07.892148 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:07.891777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:36407)
0812 14:28:07.892031 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:07.892082 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":86,"threads_started":1}
I20260812 14:28:07.892504 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:07.891884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:59385)
0812 14:28:07.892415 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:07.892457 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":86,"threads_started":1}
I20260812 14:28:07.892755 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:07.892507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:52341)
0812 14:28:07.892676 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:07.892714 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":52,"threads_started":1}
I20260812 14:28:08.893386 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:08.892980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:44743)
0812 14:28:08.893284 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:08.893332 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":60,"threads_started":1}
I20260812 14:28:08.893543 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:08.892702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:51689)
0812 14:28:08.893102 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:08.893152 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":91,"threads_started":1}
I20260812 14:28:08.893627 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:08.893350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:59051)
0812 14:28:08.893544 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:08.893585 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260812 14:28:09.894428 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:09.894137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:60545)
0812 14:28:09.894327 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:09.894370 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260812 14:28:09.894485 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:09.894137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:37031)
0812 14:28:09.894399 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:09.894435 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260812 14:28:09.894582 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:09.894358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:52397)
0812 14:28:09.894495 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:09.894540 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260812 14:28:10.895364 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:10.895039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:48221)
0812 14:28:10.895258 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:10.895301 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":72,"threads_started":1}
I20260812 14:28:10.895493 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:10.895262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:44337)
0812 14:28:10.895393 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:10.895444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260812 14:28:10.895500 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:10.895212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:51181)
0812 14:28:10.895416 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:10.895448 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":56,"threads_started":1}
I20260812 14:28:11.194542 4530 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:28:11.194654 4530 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:28:11.194787 4530 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:28:11.195174 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:11.194970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:34706)
0812 14:28:11.195082 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:11.195131 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
W20260812 14:28:11.195282 3152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:11.195809 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:11.195636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:34720)
0812 14:28:11.195742 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:11.195782 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 14:28:11.195912 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:11.195972 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:28:11.196058 4530 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:28:11.896205 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:11.895911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:41027)
0812 14:28:11.896097 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:11.896144 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260812 14:28:11.896265 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:11.896005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:43745)
0812 14:28:11.896174 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:11.896215 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260812 14:28:11.896414 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:11.896005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:49353)
0812 14:28:11.896283 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:11.896374 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":58,"threads_started":1}
I20260812 14:28:12.896982 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:12.896760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:39069)
0812 14:28:12.896915 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:12.896947 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260812 14:28:12.896965 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:12.896684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:56711)
0812 14:28:12.896846 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:12.896895 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260812 14:28:12.897161 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:12.896964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:51541)
0812 14:28:12.897071 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:12.897109 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260812 14:28:13.897900 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:13.897598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:49523)
0812 14:28:13.897792 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:13.897842 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":58,"threads_started":1}
I20260812 14:28:13.897913 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:13.897686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:36223)
0812 14:28:13.897842 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:13.897875 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260812 14:28:13.898049 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:13.897558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:55653)
0812 14:28:13.897957 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:13.898000 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":75,"threads_started":1}
I20260812 14:28:14.898943 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:14.898710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:58561)
0812 14:28:14.898862 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:14.898895 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260812 14:28:14.898944 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:14.898705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:49571)
0812 14:28:14.898845 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:14.898888 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260812 14:28:14.898944 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:14.898705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:48729)
0812 14:28:14.898849 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:14.898893 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260812 14:28:15.899729 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:15.899449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:36149)
0812 14:28:15.899601 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:15.899645 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260812 14:28:15.899753 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:15.899460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:58719)
0812 14:28:15.899638 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:15.899678 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20260812 14:28:15.899750 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:15.899542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:55867)
0812 14:28:15.899646 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:15.899677 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260812 14:28:16.413708 4556 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:28:16.413820 4556 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:28:16.413992 4556 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:28:16.414402 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:16.414206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:34910)
0812 14:28:16.414313 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:16.414352 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
W20260812 14:28:16.414486 3152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:16.414826 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:16.414736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:34922)
0812 14:28:16.414781 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:16.414807 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 14:28:16.414913 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:16.414971 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:28:16.415041 4556 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:28:16.900755 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:16.900404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:48731)
0812 14:28:16.900647 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:16.900695 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260812 14:28:16.900795 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:16.900390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:56159)
0812 14:28:16.900678 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:16.900718 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":71,"threads_started":1}
I20260812 14:28:16.901083 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:16.900855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:33237)
0812 14:28:16.901009 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:16.901042 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260812 14:28:17.901444 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:17.901193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:45019)
0812 14:28:17.901341 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:17.901391 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260812 14:28:17.901621 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:17.901437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:39929)
0812 14:28:17.901542 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:17.901576 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":41,"threads_started":1}
I20260812 14:28:17.901621 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:17.901337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:38019)
0812 14:28:17.901540 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:17.901575 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260812 14:28:18.902248 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:18.902001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:60923)
0812 14:28:18.902155 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:18.902199 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260812 14:28:18.902236 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:18.902001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:53783)
0812 14:28:18.902135 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:18.902176 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260812 14:28:18.902318 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:18.902094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:44577)
0812 14:28:18.902236 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:18.902280 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260812 14:28:19.902999 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:19.902738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.193:37499)
0812 14:28:19.902911 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:19.902943 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260812 14:28:19.902992 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:19.902738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.195:47547)
0812 14:28:19.902884 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:19.902935 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260812 14:28:19.903167 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:19.902750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.23.173.194:50963)
0812 14:28:19.903012 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:19.903056 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":67,"threads_started":1}
I20260812 14:28:20.006175 4591 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fc8af0aba90482090c23064205f5fb3)
I20260812 14:28:20.006275 4591 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fc8af0aba90482090c23064205f5fb3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46677 } } peers { permanent_uuid: "c1367ca601474caa881012adc5ea6714" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 38203 } attrs { promote: false } } peers { permanent_uuid: "b6a6c4aee9dd48cc966a5ac49e536b26" member_type: NON_VOTER last_known_addr { host: "127.23.173.252" port: 32849 } attrs { promote: true } }
I20260812 14:28:20.006400 4591 leader_election.cc:288] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677)
I20260812 14:28:20.006800 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:20.006556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:38684)
0812 14:28:20.006705 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:20.006745 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260812 14:28:20.007267 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:20.007187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46677 (local address 127.0.0.1:38698)
0812 14:28:20.007220 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:20.007248 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260812 14:28:20.007341 3152 leader_election.cc:334] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fc8af0aba90482090c23064205f5fb3 (127.23.173.254:46677): Network error: Client connection negotiation failed: client connection to 127.23.173.254:46677: connect: Connection refused (error 111)
I20260812 14:28:20.007392 3152 leader_election.cc:302] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c1367ca601474caa881012adc5ea6714; no voters: 9fc8af0aba90482090c23064205f5fb3
I20260812 14:28:20.007450 4591 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1367ca601474caa881012adc5ea6714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.23.173.254:46677,127.23.173.253:38203,127.23.173.252:32849, num_attempts: 1)
I20260812 14:28:20.078289 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 3439
I20260812 14:28:20.094628 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 3842
I20260812 14:28:20.107728 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 4086
I20260812 14:28:20.121484 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 2923
I20260812 14:28:20.134738 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 3260
2026-08-12T14:28:20Z chronyd exiting
I20260812 14:28:20.160810 24247 test_util.cc:175] -----------------------------------------------
I20260812 14:28:20.160874 24247 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544840008271-24247-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33347 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:28:20 dist-test-slave-1l3l krb5kdc[4596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:28:20 dist-test-slave-1l3l krb5kdc[4596](info): set up 2 sockets
Aug 12 14:28:20 dist-test-slave-1l3l krb5kdc[4596](info): commencing operation
krb5kdc: starting...
W20260812 14:28:21.317073 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:28:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab.
Entry for principal oryx/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab.
I20260812 14:28:21.439544 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:39855
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab
--principal=oryx/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:39855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:21.522070 4612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:21.522290 4612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:21.522325 4612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:21.523480 4612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:21.523518 4612 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:28:21.523535 4612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:21.523551 4612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:21.523566 4612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:21.523581 4612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:21.525100 4612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:39855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.254.keytab
--principal=oryx/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:39855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:21.525343 4612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:21.525568 4612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:21.530321 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:21.530416 4612 server_base.cc:1034] running on GCE node
W20260812 14:28:21.530377 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:21.530339 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:21.530821 4612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:21.531097 4612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:21.532234 4612 hybrid_clock.cc:648] HybridClock initialized: now 1786544901532220 us; error 30 us; skew 500 ppm
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:21.534535 4612 init.cc:377] Logged in from keytab as oryx/127.23.173.254@KRBTEST.COM (short username oryx)
I20260812 14:28:21.534953 4612 webserver.cc:460] Webserver started at http://127.23.173.254:39989/ using document root <none> and password file <none>
I20260812 14:28:21.535140 4612 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:21.535187 4612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:21.535310 4612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:21.536370 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "45332e6d1bdf99b96fb832eac0e5670b"
server_key_iv: "4979bc5c5da47d22259b88c52deeeaf4"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.536778 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "45332e6d1bdf99b96fb832eac0e5670b"
server_key_iv: "4979bc5c5da47d22259b88c52deeeaf4"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.538044 4612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:28:21.538671 4645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:21.538820 4612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:21.538898 4612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "45332e6d1bdf99b96fb832eac0e5670b"
server_key_iv: "4979bc5c5da47d22259b88c52deeeaf4"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.538959 4612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:21.549330 4612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:21.551040 4612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:21.551172 4612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:21.555696 4612 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:39855
I20260812 14:28:21.555783 4742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:39855 every 8 connection(s)
I20260812 14:28:21.556064 4612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:28:21.556751 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:21.559056 4744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Bootstrap starting.
I20260812 14:28:21.559638 4744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:21.559937 4744 log.cc:824] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:21.560590 4744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: No bootstrap required, opened a new log
I20260812 14:28:21.562768 4744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39855 } }
I20260812 14:28:21.562924 4744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:21.562964 4744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6585b6461cca4a8b9e6b4c3f1d7d4dc6, State: Initialized, Role: FOLLOWER
I20260812 14:28:21.563097 4744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39855 } }
I20260812 14:28:21.563159 4744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:28:21.563189 4744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:28:21.563234 4744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:21.563817 4744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39855 } }
I20260812 14:28:21.563917 4744 leader_election.cc:302] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6585b6461cca4a8b9e6b4c3f1d7d4dc6; no voters:
I20260812 14:28:21.564095 4744 leader_election.cc:288] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:28:21.564136 4749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:21.564247 4749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [term 1 LEADER]: Becoming Leader. State: Replica: 6585b6461cca4a8b9e6b4c3f1d7d4dc6, State: Running, Role: LEADER
I20260812 14:28:21.564380 4744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:28:21.564356 4749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39855 } }
I20260812 14:28:21.564821 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 4612
I20260812 14:28:21.564842 4750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39855 } } }
I20260812 14:28:21.564919 4750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:21.564936 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:28:21.565093 4754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6585b6461cca4a8b9e6b4c3f1d7d4dc6. Latest consensus state: current_term: 1 leader_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585b6461cca4a8b9e6b4c3f1d7d4dc6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39855 } } }
I20260812 14:28:21.565147 4754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:21.565171 24247 external_mini_cluster.cc:1428] Setting key 6f19044731f5b393459218c0eacf4d21
I20260812 14:28:21.565151 4762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:28:21.565775 4762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:28:21.567399 4762 catalog_manager.cc:1347] Generated new cluster ID: 8df196c56885414b92d5eaaf92765db7
I20260812 14:28:21.567454 4762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:21.573027 4762 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:28:21.573709 4762 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:28:21.574947 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.565800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.23.173.254:39855)
0812 14:28:21.566133 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:21.566143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:21.566179 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:28:21.566363 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:21.566371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:21.566434 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:21.566569 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:21.567380 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.568393 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.569100 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.569518 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.569613 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.571810 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:21.571814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:21.571814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:21.571825 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:21.573559 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.573891 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.573894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.573895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.573981 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.574157 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.574157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.574158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.574370 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:21.574529 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:21.574597 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 14:28:21.574743 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":69,"threads_started":1}
I20260812 14:28:21.577131 4762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Generated new TSK 0
I20260812 14:28:21.577288 4762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab.
Entry for principal oryx/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab.
I20260812 14:28:21.615023 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:39855
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:21.706804 4783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:21.707026 4783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:21.707051 4783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:21.708221 4783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:21.708258 4783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:21.708277 4783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:28:21.708293 4783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:28:21.709818 4783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:21.710032 4783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:21.710215 4783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:28:21.715335 4783 server_base.cc:1034] running on GCE node
W20260812 14:28:21.715358 4790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:21.715304 4791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:21.715297 4793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:21.715863 4783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:21.716133 4783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:21.717280 4783 hybrid_clock.cc:648] HybridClock initialized: now 1786544901717264 us; error 30 us; skew 500 ppm
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:21.719952 4783 init.cc:377] Logged in from keytab as oryx/127.23.173.193@KRBTEST.COM (short username oryx)
I20260812 14:28:21.720422 4783 webserver.cc:460] Webserver started at http://127.23.173.193:40603/ using document root <none> and password file <none>
I20260812 14:28:21.720652 4783 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:21.720703 4783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:21.720836 4783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:21.721999 4783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "858f8433c2dbeb5d1192c583c979c3c6"
server_key_iv: "fb1675aabbfc96b79984eb3c79807d89"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.722374 4783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "858f8433c2dbeb5d1192c583c979c3c6"
server_key_iv: "fb1675aabbfc96b79984eb3c79807d89"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.723807 4783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:21.724546 4807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:21.724711 4783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:21.724798 4783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "858f8433c2dbeb5d1192c583c979c3c6"
server_key_iv: "fb1675aabbfc96b79984eb3c79807d89"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.724870 4783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:21.748867 4783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:21.751045 4783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:21.751227 4783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:21.751844 4783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:21.751958 4783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:21.752030 4783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:21.752074 4783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:21.759213 4783 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:33319
I20260812 14:28:21.759302 4981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:33319 every 8 connection(s)
I20260812 14:28:21.759594 4783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:28:21.761996 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 4783
I20260812 14:28:21.762043 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.752397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:49611 (local address 127.23.173.254:39855)
0812 14:28:21.752450 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:21.752452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:21.752755 (+ 303us) server_negotiation.cc:407] Connection header received
0812 14:28:21.753314 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:21.753316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:21.753331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:21.753398 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:21.754255 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.755034 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.756131 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.756440 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.756523 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.759257 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:21.759259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:21.759259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:21.759268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:21.760716 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.761097 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.761098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.761099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.761158 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.761379 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.761380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.761380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.761500 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:21.761558 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:21.761848 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 14:28:21.761945 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:28:21.762109 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:28:21.762207 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.752398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.23.173.193:49611)
0812 14:28:21.752617 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:21.752658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:21.753202 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:21.753479 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:21.753482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:21.753852 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:21.754148 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.754155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.755128 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.755129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.756059 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.756062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.756542 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.756542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.756692 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.757293 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:21.757303 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:21.759145 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:21.760798 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.760806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.760808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.761027 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.761212 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.761213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.761213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.761323 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.761557 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:21.761563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:21.761705 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 14:28:21.761824 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 14:28:21.761992 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260812 14:28:21.762301 24247 external_mini_cluster.cc:1428] Setting key afa5ae19e8f1c1773bb8efa9e353e9ec
I20260812 14:28:21.762698 4982 heartbeater.cc:347] Connected to a master server at 127.23.173.254:39855
I20260812 14:28:21.762797 4982 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:21.763031 4982 heartbeater.cc:511] Master 127.23.173.254:39855 requested a full tablet report, sending...
I20260812 14:28:21.763649 4676 ts_manager.cc:194] Registered new tserver with Master: 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319)
I20260812 14:28:21.764549 4676 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:49611
I20260812 14:28:21.770118 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.763862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.23.173.193:53623)
0812 14:28:21.764024 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:21.764036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:21.764102 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:21.764217 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:21.764218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:21.764432 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:21.764507 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.764510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.765446 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.765447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.766074 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.766077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.766582 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.766583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.766666 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.767104 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:21.767109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:21.767481 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:21.769204 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.769205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.769205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.769367 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.769564 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.769565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.769565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.769618 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.769861 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:21.769861 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:21.769898 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:28:21.769946 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:28:21.770027 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260812 14:28:21.770136 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.763794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:53623 (local address 127.23.173.254:39855)
0812 14:28:21.763829 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:21.763830 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:21.764104 (+ 274us) server_negotiation.cc:407] Connection header received
0812 14:28:21.764147 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:21.764148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:21.764158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:21.764212 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:21.764577 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.765357 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.766168 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.766428 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.766489 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.767577 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:21.767579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:21.767579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:21.767585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:21.769123 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.769436 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.769437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.769438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.769498 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.769670 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.769671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.769671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.769802 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:21.769882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:21.769970 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:28:21.770045 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab.
Entry for principal oryx/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab.
I20260812 14:28:21.791831 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:39855
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:21.874148 4996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:21.874346 4996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:21.874368 4996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:21.875476 4996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:21.875512 4996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:21.875532 4996 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:28:21.875547 4996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:28:21.877135 4996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:21.877322 4996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:21.877509 4996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:21.882236 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:21.882326 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:21.882273 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:21.882447 4996 server_base.cc:1034] running on GCE node
I20260812 14:28:21.882648 4996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:21.882910 4996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:21.884038 4996 hybrid_clock.cc:648] HybridClock initialized: now 1786544901884029 us; error 29 us; skew 500 ppm
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:21.886456 4996 init.cc:377] Logged in from keytab as oryx/127.23.173.194@KRBTEST.COM (short username oryx)
I20260812 14:28:21.886888 4996 webserver.cc:460] Webserver started at http://127.23.173.194:42617/ using document root <none> and password file <none>
I20260812 14:28:21.887120 4996 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:21.887168 4996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:21.887295 4996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:21.888373 4996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "9817e4cf002447318b6a6b0108744db9"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "4227d4f75b20fa5b901ef1e88900562f"
server_key_iv: "e4110ee3900933f399d2f58f51ca0493"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.888778 4996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "9817e4cf002447318b6a6b0108744db9"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "4227d4f75b20fa5b901ef1e88900562f"
server_key_iv: "e4110ee3900933f399d2f58f51ca0493"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.890247 4996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:28:21.890877 5024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:21.891036 4996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:21.891120 4996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "9817e4cf002447318b6a6b0108744db9"
format_stamp: "Formatted at 2026-08-12 14:28:21 on dist-test-slave-1l3l"
server_key: "4227d4f75b20fa5b901ef1e88900562f"
server_key_iv: "e4110ee3900933f399d2f58f51ca0493"
server_key_version: "encryptionkey@0"
I20260812 14:28:21.891180 4996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:21.898228 4996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:21.900293 4996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:21.900432 4996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:21.900951 4996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:21.901000 4996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:21.901031 4996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:21.901058 4996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:21.908138 4996 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:41999
Aug 12 14:28:21 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:21.908586 4996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:28:21.908595 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:41999 every 8 connection(s)
I20260812 14:28:21.911379 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.901344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:33927 (local address 127.23.173.254:39855)
0812 14:28:21.901413 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:21.901416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:21.901649 (+ 233us) server_negotiation.cc:407] Connection header received
0812 14:28:21.902286 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:21.902287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:21.902301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:21.902355 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:21.903185 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.903972 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.905448 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.905669 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.905720 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.908974 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:21.908976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:21.908977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:21.908984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:21.910137 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.910533 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.910534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.910535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.910597 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.910828 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.910830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.910830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.910974 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:21.911042 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:21.911244 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 14:28:21.911312 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:28:21.911504 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.901331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.23.173.194:33927)
0812 14:28:21.901509 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:21.901557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:21.902172 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:21.902449 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:21.902453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:21.902813 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:21.903097 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.903105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.904069 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.904070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.905348 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.905352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.905742 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.905743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.905889 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.907041 (+ 1152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:21.907050 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:21.908859 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:21.910209 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.910215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.910217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.910424 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.910658 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.910658 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.910659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.910749 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.911035 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:21.911038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:21.911145 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 14:28:21.911221 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:28:21.911336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260812 14:28:21.912017 5208 heartbeater.cc:347] Connected to a master server at 127.23.173.254:39855
I20260812 14:28:21.912094 5208 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:21.912276 5208 heartbeater.cc:511] Master 127.23.173.254:39855 requested a full tablet report, sending...
I20260812 14:28:21.912701 4677 ts_manager.cc:194] Registered new tserver with Master: 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999)
I20260812 14:28:21.913193 4677 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:33927
I20260812 14:28:21.917331 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 4996
I20260812 14:28:21.917435 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:28:21.917630 24247 external_mini_cluster.cc:1428] Setting key 680dfedd710ad071ba34dbc2a32a7c05
I20260812 14:28:21.918059 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.912945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.23.173.194:43135)
0812 14:28:21.913070 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:21.913080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:21.913129 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:21.913241 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:21.913242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:21.913444 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:21.913517 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.913519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.914141 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.914142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.914652 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:21.914653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.914976 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.914977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:21.915037 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.915496 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:21.915501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:21.915802 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:21.917097 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.917098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.917098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.917258 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.917444 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:21.917445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:21.917445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:21.917506 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:21.917764 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:21.917765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:21.917814 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 14:28:21.917893 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:28:21.917989 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260812 14:28:21.918084 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:21.913023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:43135 (local address 127.23.173.254:39855)
0812 14:28:21.913074 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:21.913076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:21.913118 (+ 42us) server_negotiation.cc:407] Connection header received
0812 14:28:21.913166 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:21.913167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:21.913176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:21.913238 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:21.913587 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.914055 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.914721 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:21.914916 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:21.914970 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:21.915886 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:21.915887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:21.915888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:21.915894 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:21.917026 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.917320 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.917321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.917322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.917374 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:21.917554 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:21.917555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:21.917555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:21.917701 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:21.917770 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:21.917918 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 14:28:21.917996 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for oryx/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab.
Entry for principal oryx/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab.
I20260812 14:28:21.942488 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:39855
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:22.027282 5226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:22.027490 5226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:22.027521 5226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:22.028731 5226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:22.028776 5226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:22.028803 5226 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:28:22.028826 5226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:28:22.030313 5226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.254:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:22.030542 5226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:22.030759 5226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:22.035660 5237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:22.035684 5226 server_base.cc:1034] running on GCE node
W20260812 14:28:22.035634 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:22.035624 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:22.036128 5226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:22.036350 5226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:22.037477 5226 hybrid_clock.cc:648] HybridClock initialized: now 1786544902037467 us; error 28 us; skew 500 ppm
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:22.039929 5226 init.cc:377] Logged in from keytab as oryx/127.23.173.195@KRBTEST.COM (short username oryx)
I20260812 14:28:22.040349 5226 webserver.cc:460] Webserver started at http://127.23.173.195:43329/ using document root <none> and password file <none>
I20260812 14:28:22.040537 5226 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:22.040580 5226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:22.040706 5226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:22.041733 5226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "f7e8ba71830846f581d50b39e1e68604"
format_stamp: "Formatted at 2026-08-12 14:28:22 on dist-test-slave-1l3l"
server_key: "80f17292a88fd93333288a935e880dba"
server_key_iv: "1df379b0e37620385cf5c7c62dd5eac9"
server_key_version: "encryptionkey@0"
I20260812 14:28:22.042143 5226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "f7e8ba71830846f581d50b39e1e68604"
format_stamp: "Formatted at 2026-08-12 14:28:22 on dist-test-slave-1l3l"
server_key: "80f17292a88fd93333288a935e880dba"
server_key_iv: "1df379b0e37620385cf5c7c62dd5eac9"
server_key_version: "encryptionkey@0"
I20260812 14:28:22.043533 5226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:22.044238 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.044376 5226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.044453 5226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "f7e8ba71830846f581d50b39e1e68604"
format_stamp: "Formatted at 2026-08-12 14:28:22 on dist-test-slave-1l3l"
server_key: "80f17292a88fd93333288a935e880dba"
server_key_iv: "1df379b0e37620385cf5c7c62dd5eac9"
server_key_version: "encryptionkey@0"
I20260812 14:28:22.044518 5226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:22.058321 5226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:22.060242 5226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:22.060371 5226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:22.060843 5226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:22.060891 5226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.060920 5226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:22.060937 5226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:22.070454 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.061277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:48375 (local address 127.23.173.254:39855)
0812 14:28:22.061330 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.061332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.061536 (+ 204us) server_negotiation.cc:407] Connection header received
0812 14:28:22.062049 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.062051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.062065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.062138 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.062907 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.063521 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.064768 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.065016 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.065067 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.067420 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:22.067422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:22.067422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:22.067431 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:22.069078 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.069470 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.069478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.069479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.069549 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.069809 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.069811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.069811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.069932 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:22.069994 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.070267 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.070364 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:28:22.070605 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.061283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.23.173.195:48375)
0812 14:28:22.061427 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.061463 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.061949 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.062213 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.062217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.062526 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:22.062800 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.062807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.063605 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.063606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.064689 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.064692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.065100 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.065101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.065259 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.065905 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:22.065916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:22.067317 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:22.069157 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:22.069164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:22.069166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:22.069406 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:22.069618 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:22.069618 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:22.069619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:22.069739 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:22.069993 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:22.070000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:22.070143 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.070234 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.070388 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260812 14:28:22.070717 5226 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:34875
I20260812 14:28:22.070789 5432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:34875 every 8 connection(s)
I20260812 14:28:22.071110 5226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:28:22.071739 5433 heartbeater.cc:347] Connected to a master server at 127.23.173.254:39855
I20260812 14:28:22.071818 5433 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:22.071992 5433 heartbeater.cc:511] Master 127.23.173.254:39855 requested a full tablet report, sending...
I20260812 14:28:22.072328 4677 ts_manager.cc:194] Registered new tserver with Master: f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875)
I20260812 14:28:22.072717 4677 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:48375
I20260812 14:28:22.077278 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.072175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:39195 (local address 127.23.173.254:39855)
0812 14:28:22.072209 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.072211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.072310 (+ 99us) server_negotiation.cc:407] Connection header received
0812 14:28:22.072352 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.072352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.072362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.072408 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.072710 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.073192 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.074093 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.074274 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.074328 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.075094 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:22.075096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:22.075096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:22.075103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:22.076384 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.076666 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.076667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.076667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.076711 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.076841 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.076841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.076842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.076931 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:22.076977 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.077105 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.077178 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:28:22.077291 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.072128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.23.173.195:39195)
0812 14:28:22.072263 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.072273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.072319 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.072420 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.072420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.072600 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:22.072651 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.072653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.073267 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.073267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.074015 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.074018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.074333 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.074334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.074377 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.074741 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:22.074745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:22.075018 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:22.076458 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:22.076459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:22.076460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:22.076615 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:22.076754 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:22.076754 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:22.076755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:22.076798 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:22.076972 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:22.076973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:22.077013 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.077081 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.077192 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260812 14:28:22.080510 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 5226
I20260812 14:28:22.080607 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:28:22.080787 24247 external_mini_cluster.cc:1428] Setting key aadb58b882a5f3191902a0b974a22790
I20260812 14:28:22.081537 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:28:22.102550 4775 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:22.097682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.23.173.254:39855)
0812 14:28:22.097736 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.097738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.097746 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:28:22.097768 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.097768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.097777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.097823 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.098315 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.098783 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.099525 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.099726 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.099786 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.102463 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42712: BlockingRecv error: recv got EOF from 127.0.0.1:42712 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:22.109364 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.103298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.23.173.254:39855)
0812 14:28:22.103354 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.103356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.103421 (+ 65us) server_negotiation.cc:407] Connection header received
0812 14:28:22.103500 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.103501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.103511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.103557 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.104043 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.104494 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.105129 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.105308 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.105361 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.107280 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:22.107281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:22.107281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:22.107288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:22.108410 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.108704 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.108705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.108705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.108755 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.108918 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.108919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.108919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.109019 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:22.109103 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.109203 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.109279 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:28:22.110754 4677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:28:22.112481 4677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:28:22.118901 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.114658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:34875 (local address 127.0.0.1:36908)
0812 14:28:22.114811 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.114846 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.114911 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.115486 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.115487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.115489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.115525 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.115527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.116750 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.116751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.117750 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.117752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.118607 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.118701 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.118718 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.118791 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.118818 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":23,"thread_start_us":45,"threads_started":1}
I20260812 14:28:22.118966 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.114908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.23.173.195:34875)
0812 14:28:22.115286 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.115290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.115299 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:22.115335 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.115351 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.115365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.115500 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.115569 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.116663 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.117825 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.118520 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118614 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.118673 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.118814 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.118863 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":89,"threads_started":1}
I20260812 14:28:22.119136 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.114947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.23.173.194:41999)
0812 14:28:22.115114 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.115120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.115132 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:28:22.115172 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.115173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.115189 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.115272 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.115452 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.116625 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118075 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.118569 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118886 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.118951 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.118991 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.119034 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260812 14:28:22.119241 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.114658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33319 (local address 127.0.0.1:48006)
0812 14:28:22.115794 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.115803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.115854 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.116006 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.116007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.116008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.116045 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.116047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.116937 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.116938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.118002 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.118004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.118981 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.119044 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.119048 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.119126 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.119175 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260812 14:28:22.119289 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.115075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.23.173.193:33319)
0812 14:28:22.115223 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.115227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.115867 (+ 640us) server_negotiation.cc:407] Connection header received
0812 14:28:22.115917 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.115920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.115935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.116004 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.116097 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.116851 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118078 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.118789 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.118941 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.119151 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.119192 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260812 14:28:22.119670 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.114657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41999 (local address 127.0.0.1:58964)
0812 14:28:22.114803 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.114846 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.114911 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.115296 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.115299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.115302 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.115368 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.115370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.116728 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.116729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.117946 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.117949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.118658 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.118659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.118708 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.118719 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.118765 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.118814 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":95,"threads_started":1}
I20260812 14:28:22.120093 5324 tablet_service.cc:1467] Processing CreateTablet for tablet b42d4afb49c94614a5bf5c3b50076365 (DEFAULT_TABLE table=test-table [id=1393f66ebc7e4fd1a674e69a2ef81198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.120479 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b42d4afb49c94614a5bf5c3b50076365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.120388 5106 tablet_service.cc:1467] Processing CreateTablet for tablet b42d4afb49c94614a5bf5c3b50076365 (DEFAULT_TABLE table=test-table [id=1393f66ebc7e4fd1a674e69a2ef81198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.120682 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b42d4afb49c94614a5bf5c3b50076365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.123517 5478 tablet_bootstrap.cc:492] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Bootstrap starting.
I20260812 14:28:22.123972 5479 tablet_bootstrap.cc:492] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Bootstrap starting.
I20260812 14:28:22.124012 4878 tablet_service.cc:1467] Processing CreateTablet for tablet b42d4afb49c94614a5bf5c3b50076365 (DEFAULT_TABLE table=test-table [id=1393f66ebc7e4fd1a674e69a2ef81198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.124269 5478 tablet_bootstrap.cc:654] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.124362 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b42d4afb49c94614a5bf5c3b50076365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.124568 5479 tablet_bootstrap.cc:654] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.124636 5478 log.cc:824] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:22.124862 5479 log.cc:824] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:22.125367 5478 tablet_bootstrap.cc:492] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: No bootstrap required, opened a new log
I20260812 14:28:22.125442 5478 ts_tablet_manager.cc:1397] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:28:22.125664 5479 tablet_bootstrap.cc:492] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: No bootstrap required, opened a new log
I20260812 14:28:22.125736 5479 ts_tablet_manager.cc:1397] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:28:22.127463 5478 raft_consensus.cc:348] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.127638 5478 raft_consensus.cc:374] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.127672 5478 raft_consensus.cc:729] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.127840 5478 consensus_queue.cc:260] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.127779 5479 raft_consensus.cc:348] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.127925 5479 raft_consensus.cc:374] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.127967 5479 raft_consensus.cc:729] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817e4cf002447318b6a6b0108744db9, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.128119 5478 ts_tablet_manager.cc:1428] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 14:28:22.128095 5479 consensus_queue.cc:260] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.128206 5485 tablet_bootstrap.cc:492] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Bootstrap starting.
I20260812 14:28:22.128253 5433 heartbeater.cc:503] Master 127.23.173.254:39855 was elected leader, sending a full tablet report...
I20260812 14:28:22.128347 5208 heartbeater.cc:503] Master 127.23.173.254:39855 was elected leader, sending a full tablet report...
I20260812 14:28:22.128333 5479 ts_tablet_manager.cc:1428] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:28:22.128911 5485 tablet_bootstrap.cc:654] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.129333 5485 log.cc:824] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:22.130041 5485 tablet_bootstrap.cc:492] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: No bootstrap required, opened a new log
I20260812 14:28:22.130096 5485 ts_tablet_manager.cc:1397] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:28:22.131605 5485 raft_consensus.cc:348] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.131757 5485 raft_consensus.cc:374] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.131799 5485 raft_consensus.cc:729] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851d1ae37b1e4910bcc6be1dd9f4a8b0, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.131916 5485 consensus_queue.cc:260] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.132153 5485 ts_tablet_manager.cc:1428] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:28:22.132190 4982 heartbeater.cc:503] Master 127.23.173.254:39855 was elected leader, sending a full tablet report...
W20260812 14:28:22.159221 5209 tablet.cc:2369] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:28:22.172315 5486 raft_consensus.cc:482] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:28:22.172474 5486 raft_consensus.cc:504] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.172840 5486 leader_election.cc:288] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999), 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319)
I20260812 14:28:22.175609 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.173141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:58035 (local address 127.23.173.193:33319)
0812 14:28:22.173219 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.173221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.173228 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:28:22.173254 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.173254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.173262 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.173311 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.173516 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.173991 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.174892 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.175260 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.175323 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.175332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.175501 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.175544 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 14:28:22.175631 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.173016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33319 (local address 127.23.173.195:58035)
0812 14:28:22.173073 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.173085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.173128 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.173363 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.173364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.173366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.173424 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.173426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.174071 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.174072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.174803 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.174805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.175333 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.175334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.175429 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.175432 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.175479 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.175519 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:28:22.175639 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.173140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:59321 (local address 127.23.173.194:41999)
0812 14:28:22.173185 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.173187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.173310 (+ 123us) server_negotiation.cc:407] Connection header received
0812 14:28:22.173344 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.173345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.173354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.173409 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.173562 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.174023 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.174926 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.175306 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.175384 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.175393 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.175530 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.175571 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:28:22.175659 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.173029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41999 (local address 127.23.173.195:59321)
0812 14:28:22.173243 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.173251 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.173306 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.173417 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.173418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.173420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.173502 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.173504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.174107 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.174108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.174843 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.174845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.175384 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.175385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.175469 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.175472 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.175518 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.175566 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":133,"threads_started":1}
I20260812 14:28:22.176098 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42d4afb49c94614a5bf5c3b50076365" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9" is_pre_election: true
I20260812 14:28:22.176100 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42d4afb49c94614a5bf5c3b50076365" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" is_pre_election: true
I20260812 14:28:22.176239 4910 raft_consensus.cc:2454] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 0.
I20260812 14:28:22.176244 5140 raft_consensus.cc:2454] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 0.
I20260812 14:28:22.176477 5249 leader_election.cc:302] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, 9817e4cf002447318b6a6b0108744db9, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.176654 5486 raft_consensus.cc:2781] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:28:22.176713 5486 raft_consensus.cc:482] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:28:22.176736 5486 raft_consensus.cc:3037] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.177661 5486 raft_consensus.cc:504] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.177796 5486 leader_election.cc:288] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 election: Requested vote from peers 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999), 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319)
I20260812 14:28:22.177950 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42d4afb49c94614a5bf5c3b50076365" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
I20260812 14:28:22.177975 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42d4afb49c94614a5bf5c3b50076365" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9"
I20260812 14:28:22.178045 5140 raft_consensus.cc:3037] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.178045 4910 raft_consensus.cc:3037] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.178920 5140 raft_consensus.cc:2454] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 1.
I20260812 14:28:22.178920 4910 raft_consensus.cc:2454] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 1.
I20260812 14:28:22.179098 5254 leader_election.cc:302] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.179220 5486 raft_consensus.cc:2781] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:22.179361 5486 raft_consensus.cc:686] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 1 LEADER]: Becoming Leader. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Running, Role: LEADER
I20260812 14:28:22.179438 5486 consensus_queue.cc:237] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.180279 4676 catalog_manager.cc:5495] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7e8ba71830846f581d50b39e1e68604 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "f7e8ba71830846f581d50b39e1e68604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } health_report { overall_health: HEALTHY } } }
I20260812 14:28:22.192309 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.190005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.23.173.195:34875)
0812 14:28:22.190068 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.190070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.190079 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:22.190113 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.190114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.190132 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.190190 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.190325 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.190808 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.191536 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.191773 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.191824 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.191991 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.192100 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.192172 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.192210 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.192232 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:28:22.194384 4910 raft_consensus.cc:1261] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Refusing update from remote peer f7e8ba71830846f581d50b39e1e68604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.194401 5140 raft_consensus.cc:1261] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Refusing update from remote peer f7e8ba71830846f581d50b39e1e68604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.194746 5493 consensus_queue.cc:1035] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.194877 5486 consensus_queue.cc:1035] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.197441 5499 mvcc.cc:205] Tried to move back new op lower bound from 7317687919385608192 to 7317687919327674368. Current Snapshot: MvccSnapshot[applied={T|T < 7317687919385608192}]
I20260812 14:28:22.197330 5497 mvcc.cc:205] Tried to move back new op lower bound from 7317687919385608192 to 7317687919327674368. Current Snapshot: MvccSnapshot[applied={T|T < 7317687919385608192}]
I20260812 14:28:22.198611 5502 mvcc.cc:205] Tried to move back new op lower bound from 7317687919385608192 to 7317687919327674368. Current Snapshot: MvccSnapshot[applied={T|T < 7317687919385608192 or (T in {7317687919385608192})}]
I20260812 14:28:22.200030 4676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.200121 4676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.200829 4676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Sending DeleteTablet for 3 replicas of tablet b42d4afb49c94614a5bf5c3b50076365
I20260812 14:28:22.201152 4878 tablet_service.cc:1514] Processing DeleteTablet for tablet b42d4afb49c94614a5bf5c3b50076365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 14:28:22.201354 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet b42d4afb49c94614a5bf5c3b50076365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:58964
I20260812 14:28:22.201582 5512 tablet_replica.cc:330] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: stopping tablet replica
I20260812 14:28:22.201691 5512 raft_consensus.cc:2229] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.201686 5324 tablet_service.cc:1514] Processing DeleteTablet for tablet b42d4afb49c94614a5bf5c3b50076365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36908
I20260812 14:28:22.201825 5512 raft_consensus.cc:2258] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.201956 4676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:28:22.202121 4676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:28:22.202035 5513 tablet_replica.cc:330] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: stopping tablet replica
I20260812 14:28:22.202332 5512 ts_tablet_manager.cc:1905] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.202338 5513 raft_consensus.cc:2229] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:28:22.202517 5513 raft_consensus.cc:2258] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.203042 5513 ts_tablet_manager.cc:1905] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.203457 4878 tablet_service.cc:1467] Processing CreateTablet for tablet f3d1482774b3415a93d71013023d6e52 (DEFAULT_TABLE table=test-table [id=437c62c725d848ce98800dfaeed87668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.203586 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d1482774b3415a93d71013023d6e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.204041 5512 ts_tablet_manager.cc:1918] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:22.204120 5512 log.cc:1196] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/b42d4afb49c94614a5bf5c3b50076365
I20260812 14:28:22.204385 5512 ts_tablet_manager.cc:1939] T b42d4afb49c94614a5bf5c3b50076365 P 9817e4cf002447318b6a6b0108744db9: Deleting consensus metadata
I20260812 14:28:22.205080 5513 ts_tablet_manager.cc:1918] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:22.205201 5485 tablet_bootstrap.cc:492] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Bootstrap starting.
I20260812 14:28:22.205221 5513 log.cc:1196] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/b42d4afb49c94614a5bf5c3b50076365
I20260812 14:28:22.205528 5324 tablet_service.cc:1467] Processing CreateTablet for tablet f3d1482774b3415a93d71013023d6e52 (DEFAULT_TABLE table=test-table [id=437c62c725d848ce98800dfaeed87668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.205636 5485 tablet_bootstrap.cc:654] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.205636 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d1482774b3415a93d71013023d6e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.205768 4647 catalog_manager.cc:4841] TS 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999): tablet b42d4afb49c94614a5bf5c3b50076365 (table test-table [id=1393f66ebc7e4fd1a674e69a2ef81198]) successfully deleted
I20260812 14:28:22.206192 5106 tablet_service.cc:1467] Processing CreateTablet for tablet f3d1482774b3415a93d71013023d6e52 (DEFAULT_TABLE table=test-table [id=437c62c725d848ce98800dfaeed87668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.206269 5485 tablet_bootstrap.cc:492] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: No bootstrap required, opened a new log
I20260812 14:28:22.206290 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d1482774b3415a93d71013023d6e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.206310 5485 ts_tablet_manager.cc:1397] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:22.206454 5485 raft_consensus.cc:348] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.206516 5485 raft_consensus.cc:374] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.206532 5485 raft_consensus.cc:729] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851d1ae37b1e4910bcc6be1dd9f4a8b0, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.206570 5485 consensus_queue.cc:260] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.206645 5485 ts_tablet_manager.cc:1428] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.205571 5513 ts_tablet_manager.cc:1939] T b42d4afb49c94614a5bf5c3b50076365 P f7e8ba71830846f581d50b39e1e68604: Deleting consensus metadata
I20260812 14:28:22.207396 5479 tablet_bootstrap.cc:492] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Bootstrap starting.
I20260812 14:28:22.207409 5478 tablet_bootstrap.cc:492] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Bootstrap starting.
I20260812 14:28:22.207547 4648 catalog_manager.cc:4841] TS f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875): tablet b42d4afb49c94614a5bf5c3b50076365 (table test-table [id=1393f66ebc7e4fd1a674e69a2ef81198]) successfully deleted
I20260812 14:28:22.207826 5479 tablet_bootstrap.cc:654] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.207826 5478 tablet_bootstrap.cc:654] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.208294 5511 tablet_replica.cc:330] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: stopping tablet replica
I20260812 14:28:22.208421 5511 raft_consensus.cc:2229] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.208487 5478 tablet_bootstrap.cc:492] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: No bootstrap required, opened a new log
I20260812 14:28:22.208537 5478 ts_tablet_manager.cc:1397] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:22.208545 5511 raft_consensus.cc:2258] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.208696 5478 raft_consensus.cc:348] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.208777 5478 raft_consensus.cc:374] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.208796 5478 raft_consensus.cc:729] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.208835 5478 consensus_queue.cc:260] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.208945 5478 ts_tablet_manager.cc:1428] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.208986 5511 ts_tablet_manager.cc:1905] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.209331 5479 tablet_bootstrap.cc:492] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: No bootstrap required, opened a new log
I20260812 14:28:22.209372 5479 ts_tablet_manager.cc:1397] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:28:22.209527 5479 raft_consensus.cc:348] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.209594 5479 raft_consensus.cc:374] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.209612 5479 raft_consensus.cc:729] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817e4cf002447318b6a6b0108744db9, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.209650 5479 consensus_queue.cc:260] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.209730 5479 ts_tablet_manager.cc:1428] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.210453 5511 ts_tablet_manager.cc:1918] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:22.210538 5511 log.cc:1196] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/b42d4afb49c94614a5bf5c3b50076365
I20260812 14:28:22.210783 5511 ts_tablet_manager.cc:1939] T b42d4afb49c94614a5bf5c3b50076365 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting consensus metadata
I20260812 14:28:22.211249 4650 catalog_manager.cc:4841] TS 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319): tablet b42d4afb49c94614a5bf5c3b50076365 (table test-table [id=1393f66ebc7e4fd1a674e69a2ef81198]) successfully deleted
W20260812 14:28:22.260365 4986 tablet.cc:2369] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:28:22.321867 5434 tablet.cc:2369] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:28:22.343874 5491 raft_consensus.cc:482] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:28:22.343997 5491 raft_consensus.cc:504] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.344364 5491 leader_election.cc:288] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999), f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875)
I20260812 14:28:22.347457 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.344635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51381 (local address 127.23.173.195:34875)
0812 14:28:22.344691 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.344693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.344854 (+ 161us) server_negotiation.cc:407] Connection header received
0812 14:28:22.344957 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.344959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.344984 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.345039 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.345191 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.345695 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.346696 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.347099 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.347155 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.347163 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.347340 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.347386 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:28:22.347466 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.344708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:48649 (local address 127.23.173.194:41999)
0812 14:28:22.344748 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.344750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.344815 (+ 65us) server_negotiation.cc:407] Connection header received
0812 14:28:22.344898 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.344899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.344909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.344972 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.345142 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.345710 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.346698 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.347150 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.347222 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.347231 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.347357 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.347400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:28:22.347496 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.344573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:34875 (local address 127.23.173.193:51381)
0812 14:28:22.344784 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.344794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.344874 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.345054 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.345055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.345057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.345121 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.345123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.345782 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.345783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.346600 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.346602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.347173 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.347174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.347247 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.347252 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.347317 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.347363 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":94,"threads_started":1}
I20260812 14:28:22.347499 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.344572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41999 (local address 127.23.173.193:48649)
0812 14:28:22.344752 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.344762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.344825 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.344984 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.344985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.344988 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.345075 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.345077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.345798 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.345799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.346604 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.346607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.347231 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.347232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.347288 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.347292 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.347339 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.347382 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":48,"thread_start_us":96,"threads_started":1}
I20260812 14:28:22.347761 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d1482774b3415a93d71013023d6e52" candidate_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8ba71830846f581d50b39e1e68604" is_pre_election: true
I20260812 14:28:22.347892 5360 raft_consensus.cc:2454] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851d1ae37b1e4910bcc6be1dd9f4a8b0 in term 0.
I20260812 14:28:22.348085 4810 leader_election.cc:302] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.348083 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d1482774b3415a93d71013023d6e52" candidate_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9" is_pre_election: true
I20260812 14:28:22.348179 5140 raft_consensus.cc:2454] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851d1ae37b1e4910bcc6be1dd9f4a8b0 in term 0.
I20260812 14:28:22.348258 5491 raft_consensus.cc:2781] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:28:22.348322 5491 raft_consensus.cc:482] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:28:22.348348 5491 raft_consensus.cc:3037] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.349045 5491 raft_consensus.cc:504] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.349172 5491 leader_election.cc:288] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [CANDIDATE]: Term 1 election: Requested vote from peers 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999), f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875)
I20260812 14:28:22.349336 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d1482774b3415a93d71013023d6e52" candidate_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8ba71830846f581d50b39e1e68604"
I20260812 14:28:22.349426 5360 raft_consensus.cc:3037] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.349412 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d1482774b3415a93d71013023d6e52" candidate_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9"
I20260812 14:28:22.349511 5140 raft_consensus.cc:3037] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.350247 5360 raft_consensus.cc:2454] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851d1ae37b1e4910bcc6be1dd9f4a8b0 in term 1.
I20260812 14:28:22.350246 5140 raft_consensus.cc:2454] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851d1ae37b1e4910bcc6be1dd9f4a8b0 in term 1.
I20260812 14:28:22.350401 4810 leader_election.cc:302] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.350494 5491 raft_consensus.cc:2781] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:22.350643 5491 raft_consensus.cc:686] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 LEADER]: Becoming Leader. State: Replica: 851d1ae37b1e4910bcc6be1dd9f4a8b0, State: Running, Role: LEADER
I20260812 14:28:22.350703 5491 consensus_queue.cc:237] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.351366 4676 catalog_manager.cc:5495] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193). New cstate: current_term: 1 leader_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } health_report { overall_health: UNKNOWN } } }
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.254@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:22.386770 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.380478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:39855 (local address 127.0.0.1:42744)
0812 14:28:22.380642 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.380654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.380725 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.380854 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.380855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.381191 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:22.381242 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.381245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.381878 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.381879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.382392 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.382393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.382724 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.382725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.382773 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.383238 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:22.383242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:22.384551 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:22.385782 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:22.385790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:22.385792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:22.385985 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:22.386152 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:22.386152 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:22.386153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:22.386202 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:22.386404 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:22.386412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:22.386457 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.386515 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.386631 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260812 14:28:22.386812 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.380546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.23.173.254:39855)
0812 14:28:22.380612 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.380614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.380719 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:28:22.380777 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.380778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.380789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.380845 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.381327 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.381809 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.382492 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.382665 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.382720 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.384645 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:22.384646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:22.384647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:22.384653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:22.385710 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.386050 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.386052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.386052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.386099 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.386251 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.386252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.386252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.386351 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:22.386405 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.386545 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.386693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":35}
I20260812 14:28:22.388535 4676 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:42744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:28:22.388747 4676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:28:22.390097 5324 tablet_service.cc:1467] Processing CreateTablet for tablet 748e4f160bf7463cb5b0f069a88f0a16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88d4bd3f349843fb84865d3ca3312dee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:28:22.390113 4878 tablet_service.cc:1467] Processing CreateTablet for tablet 748e4f160bf7463cb5b0f069a88f0a16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88d4bd3f349843fb84865d3ca3312dee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:28:22.390113 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 748e4f160bf7463cb5b0f069a88f0a16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88d4bd3f349843fb84865d3ca3312dee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:28:22.390257 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748e4f160bf7463cb5b0f069a88f0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.390262 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748e4f160bf7463cb5b0f069a88f0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.390262 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748e4f160bf7463cb5b0f069a88f0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.391464 5478 tablet_bootstrap.cc:492] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: Bootstrap starting.
I20260812 14:28:22.391811 5479 tablet_bootstrap.cc:492] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: Bootstrap starting.
I20260812 14:28:22.391940 5478 tablet_bootstrap.cc:654] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.391993 5485 tablet_bootstrap.cc:492] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Bootstrap starting.
I20260812 14:28:22.392244 5479 tablet_bootstrap.cc:654] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.392386 5485 tablet_bootstrap.cc:654] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.392582 5478 tablet_bootstrap.cc:492] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: No bootstrap required, opened a new log
I20260812 14:28:22.392632 5478 ts_tablet_manager.cc:1397] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:22.392757 5478 raft_consensus.cc:348] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.392832 5478 raft_consensus.cc:374] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.392843 5479 tablet_bootstrap.cc:492] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: No bootstrap required, opened a new log
I20260812 14:28:22.392851 5478 raft_consensus.cc:729] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.392881 5479 ts_tablet_manager.cc:1397] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:22.392892 5478 consensus_queue.cc:260] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.392999 5478 ts_tablet_manager.cc:1428] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:28:22.393049 5479 raft_consensus.cc:348] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.393029 5507 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.393108 5479 raft_consensus.cc:374] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.393127 5479 raft_consensus.cc:729] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817e4cf002447318b6a6b0108744db9, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.393126 5507 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.393169 5479 consensus_queue.cc:260] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.393198 5485 tablet_bootstrap.cc:492] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: No bootstrap required, opened a new log
I20260812 14:28:22.393234 5485 ts_tablet_manager.cc:1397] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:22.393250 5479 ts_tablet_manager.cc:1428] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.393376 5485 raft_consensus.cc:348] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.393448 5485 raft_consensus.cc:374] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.393405 5487 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.393467 5485 raft_consensus.cc:729] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851d1ae37b1e4910bcc6be1dd9f4a8b0, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.393481 5487 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.393504 5485 consensus_queue.cc:260] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.393582 5485 ts_tablet_manager.cc:1428] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.393676 5491 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.393730 5491 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.464772 5487 raft_consensus.cc:482] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:28:22.464890 5487 raft_consensus.cc:504] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.465258 5487 leader_election.cc:288] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319), f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875)
I20260812 14:28:22.469537 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.465732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:46151 (local address 127.23.173.195:34875)
0812 14:28:22.465776 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.465778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.465789 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:28:22.465819 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.465820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.465829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.465895 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.466064 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.466844 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.468505 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.469088 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.469170 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.469180 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.469397 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.469451 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:28:22.469529 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.465663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:57599 (local address 127.23.173.193:33319)
0812 14:28:22.465711 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.465714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.465880 (+ 166us) server_negotiation.cc:407] Connection header received
0812 14:28:22.465932 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.465933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.465943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.465989 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:22.466117 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.466841 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.468503 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.469095 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.469170 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.469180 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.469397 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.469446 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:28:22.469578 5543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.465468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33319 (local address 127.23.173.194:57599)
0812 14:28:22.465648 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.465797 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.465843 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.466006 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.466007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.466008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.466058 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.466059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.466975 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.466976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.468361 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.468365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.469200 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.469201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.469312 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.469318 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.469382 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.469447 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":67,"thread_start_us":72,"threads_started":1}
I20260812 14:28:22.469590 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.465467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:34875 (local address 127.23.173.194:46151)
0812 14:28:22.465636 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.465660 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.465711 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.465908 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.465909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.465910 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:22.465997 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.465999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.466975 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.466976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.468361 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.468365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.469199 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.469200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.469312 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.469319 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.469382 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.469448 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":42,"thread_start_us":80,"threads_started":1}
I20260812 14:28:22.469969 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e4f160bf7463cb5b0f069a88f0a16" candidate_uuid: "9817e4cf002447318b6a6b0108744db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" is_pre_election: true
I20260812 14:28:22.470027 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e4f160bf7463cb5b0f069a88f0a16" candidate_uuid: "9817e4cf002447318b6a6b0108744db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8ba71830846f581d50b39e1e68604" is_pre_election: true
I20260812 14:28:22.470077 4910 raft_consensus.cc:2454] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9817e4cf002447318b6a6b0108744db9 in term 0.
I20260812 14:28:22.470114 5360 raft_consensus.cc:2454] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9817e4cf002447318b6a6b0108744db9 in term 0.
I20260812 14:28:22.470278 5031 leader_election.cc:302] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, 9817e4cf002447318b6a6b0108744db9; no voters:
I20260812 14:28:22.470439 5487 raft_consensus.cc:2781] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:28:22.470485 5487 raft_consensus.cc:482] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:28:22.470507 5487 raft_consensus.cc:3037] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.471277 5487 raft_consensus.cc:504] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.471424 5487 leader_election.cc:288] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [CANDIDATE]: Term 1 election: Requested vote from peers 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319), f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875)
I20260812 14:28:22.471592 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e4f160bf7463cb5b0f069a88f0a16" candidate_uuid: "9817e4cf002447318b6a6b0108744db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
I20260812 14:28:22.471597 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e4f160bf7463cb5b0f069a88f0a16" candidate_uuid: "9817e4cf002447318b6a6b0108744db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8ba71830846f581d50b39e1e68604"
I20260812 14:28:22.471674 5360 raft_consensus.cc:3037] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.471670 4910 raft_consensus.cc:3037] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.472334 4910 raft_consensus.cc:2454] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9817e4cf002447318b6a6b0108744db9 in term 1.
I20260812 14:28:22.472373 5360 raft_consensus.cc:2454] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9817e4cf002447318b6a6b0108744db9 in term 1.
I20260812 14:28:22.472519 5031 leader_election.cc:302] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, 9817e4cf002447318b6a6b0108744db9; no voters:
I20260812 14:28:22.472620 5487 raft_consensus.cc:2781] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:22.472788 5487 raft_consensus.cc:686] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [term 1 LEADER]: Becoming Leader. State: Replica: 9817e4cf002447318b6a6b0108744db9, State: Running, Role: LEADER
I20260812 14:28:22.472856 5487 consensus_queue.cc:237] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.473222 5545 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: TxnStatusTablet state changed. Reason: New leader 9817e4cf002447318b6a6b0108744db9. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.473316 5545 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:28:22.473531 5549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:28:22.473641 5549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:28:22.473609 4677 catalog_manager.cc:5495] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9817e4cf002447318b6a6b0108744db9 (127.23.173.194). New cstate: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } health_report { overall_health: UNKNOWN } } }
I20260812 14:28:22.488941 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.486521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.23.173.194:41999)
0812 14:28:22.486572 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.486574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.486584 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:22.486660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.486661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.486680 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.486740 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.486868 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.487357 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.488163 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.488371 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.488426 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.488620 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.488723 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.488788 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.488838 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.488859 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:28:22.488951 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.486440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41999 (local address 127.0.0.1:58974)
0812 14:28:22.486512 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.486525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.486583 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.486751 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.486752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.486753 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:28:22.486805 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.486808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.487434 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.487435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.488088 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.488091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.488437 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.488438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.488512 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.488527 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.488784 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.488792 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.488835 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.488883 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:28:22.490211 4910 raft_consensus.cc:1261] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Refusing update from remote peer 9817e4cf002447318b6a6b0108744db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.490351 5360 raft_consensus.cc:1261] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Refusing update from remote peer 9817e4cf002447318b6a6b0108744db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.490499 5545 consensus_queue.cc:1035] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.490622 5487 consensus_queue.cc:1035] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.491282 5491 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: TxnStatusTablet state changed. Reason: New leader 9817e4cf002447318b6a6b0108744db9. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.491374 5491 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.491346 5507 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: TxnStatusTablet state changed. Reason: New leader 9817e4cf002447318b6a6b0108744db9. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.491421 5507 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.491962 5545 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.492043 5545 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:28:22.492440 5491 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.492520 5491 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.492884 5545 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.492959 5545 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P 9817e4cf002447318b6a6b0108744db9: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:28:22.493602 5507 tablet_replica.cc:439] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9817e4cf002447318b6a6b0108744db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } } }
I20260812 14:28:22.493690 5507 tablet_replica.cc:442] T 748e4f160bf7463cb5b0f069a88f0a16 P f7e8ba71830846f581d50b39e1e68604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:28:22.496628 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.494348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.23.173.193:33319)
0812 14:28:22.494388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.494390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.494397 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:28:22.494450 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.494451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.494474 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.494536 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.494661 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.495142 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.495910 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.496119 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.496174 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.496327 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.496434 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.496495 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.496537 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.496559 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:28:22.502182 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.499446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:39975 (local address 127.23.173.194:41999)
0812 14:28:22.499507 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.499510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.499522 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:28:22.499595 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.499596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.499603 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.499650 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.499824 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.500289 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.501447 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.501644 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.501707 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.501918 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.501953 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.502013 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.502086 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.502114 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:28:22.502216 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.499292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41999 (local address 127.23.173.193:39975)
0812 14:28:22.499457 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.499469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.499534 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.499662 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.499663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.499665 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:28:22.499766 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.499768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.500351 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.500352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.501343 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.501346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.501717 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.501718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.501814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.501834 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.502017 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.502019 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.502067 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.502115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260812 14:28:22.505134 5360 raft_consensus.cc:1261] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Refusing update from remote peer 851d1ae37b1e4910bcc6be1dd9f4a8b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.505146 5140 raft_consensus.cc:1261] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Refusing update from remote peer 851d1ae37b1e4910bcc6be1dd9f4a8b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.505584 5524 consensus_queue.cc:1035] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.505718 5491 consensus_queue.cc:1035] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.514092 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.511563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:40155 (local address 127.23.173.193:33319)
0812 14:28:22.511612 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.511613 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.511621 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:28:22.511677 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.511678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.511688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.511752 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.511912 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.512486 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.513402 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.513628 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.513687 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.513869 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.513905 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.513948 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.513997 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.514018 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:28:22.514128 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.511408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33319 (local address 127.23.173.194:40155)
0812 14:28:22.511564 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.511576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.511630 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.511793 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.511795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.511796 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:28:22.511850 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.511852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.512559 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.512560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.513346 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.513348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.513724 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.513725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.513783 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.513798 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.513955 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.513956 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.513996 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.514037 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260812 14:28:22.521118 4677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.521238 4677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.521901 4677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Sending DeleteTablet for 3 replicas of tablet f3d1482774b3415a93d71013023d6e52
I20260812 14:28:22.522217 5324 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d1482774b3415a93d71013023d6e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36908
I20260812 14:28:22.522213 4878 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d1482774b3415a93d71013023d6e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 14:28:22.522312 5511 tablet_replica.cc:330] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: stopping tablet replica
I20260812 14:28:22.522321 5513 tablet_replica.cc:330] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: stopping tablet replica
I20260812 14:28:22.522364 5511 raft_consensus.cc:2229] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:28:22.522378 5513 raft_consensus.cc:2229] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.522413 5513 raft_consensus.cc:2258] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.522464 5511 raft_consensus.cc:2258] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.522856 5511 ts_tablet_manager.cc:1905] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.524596 5511 ts_tablet_manager.cc:1918] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:28:22.524664 5511 log.cc:1196] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/f3d1482774b3415a93d71013023d6e52
I20260812 14:28:22.524875 5511 ts_tablet_manager.cc:1939] T f3d1482774b3415a93d71013023d6e52 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting consensus metadata
I20260812 14:28:22.525168 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d1482774b3415a93d71013023d6e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:58964
I20260812 14:28:22.525265 5512 tablet_replica.cc:330] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: stopping tablet replica
I20260812 14:28:22.525309 5512 raft_consensus.cc:2229] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.525343 5512 raft_consensus.cc:2258] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.525725 5512 ts_tablet_manager.cc:1905] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.527412 5512 ts_tablet_manager.cc:1918] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:28:22.527477 5512 log.cc:1196] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/f3d1482774b3415a93d71013023d6e52
I20260812 14:28:22.527729 5512 ts_tablet_manager.cc:1939] T f3d1482774b3415a93d71013023d6e52 P 9817e4cf002447318b6a6b0108744db9: Deleting consensus metadata
I20260812 14:28:22.528182 4647 catalog_manager.cc:4841] TS 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999): tablet f3d1482774b3415a93d71013023d6e52 (table test-table [id=437c62c725d848ce98800dfaeed87668]) successfully deleted
I20260812 14:28:22.529279 4650 catalog_manager.cc:4841] TS 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319): tablet f3d1482774b3415a93d71013023d6e52 (table test-table [id=437c62c725d848ce98800dfaeed87668]) successfully deleted
I20260812 14:28:22.529424 5513 ts_tablet_manager.cc:1905] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.531036 5513 ts_tablet_manager.cc:1918] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:28:22.531106 5513 log.cc:1196] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/f3d1482774b3415a93d71013023d6e52
I20260812 14:28:22.531309 5513 ts_tablet_manager.cc:1939] T f3d1482774b3415a93d71013023d6e52 P f7e8ba71830846f581d50b39e1e68604: Deleting consensus metadata
I20260812 14:28:22.531656 4648 catalog_manager.cc:4841] TS f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875): tablet f3d1482774b3415a93d71013023d6e52 (table test-table [id=437c62c725d848ce98800dfaeed87668]) successfully deleted
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:28:22.562809 4775 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:22.552457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.23.173.254:39855)
0812 14:28:22.552507 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.552510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.552825 (+ 315us) server_negotiation.cc:407] Connection header received
0812 14:28:22.552872 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.552873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.552887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.552954 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.553677 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.554421 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.555579 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.555925 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.555976 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.562740 (+ 6764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42756: BlockingRecv error: recv got EOF from 127.0.0.1:42756 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[4596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM
I20260812 14:28:22.573354 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.563971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.23.173.254:39855)
0812 14:28:22.564134 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.564137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.564506 (+ 369us) server_negotiation.cc:407] Connection header received
0812 14:28:22.564566 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.564568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.564581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.564648 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:22.565242 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.566039 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.567099 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.567396 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.567480 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.570260 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:22.570262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:22.570262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:22.570270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:22.572066 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.572529 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.572530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.572531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.572596 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:22.572800 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:22.572801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:22.572802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:22.572936 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:22.573033 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.573168 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.573254 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260812 14:28:22.575256 4677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:28:22.575470 4677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:28:22.577157 4878 tablet_service.cc:1467] Processing CreateTablet for tablet 84f32fa567444ffa95c85ccf40b8f9b1 (DEFAULT_TABLE table=test-table [id=ef42e4cf00704b32b1f1f0c13eed898c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.577201 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 84f32fa567444ffa95c85ccf40b8f9b1 (DEFAULT_TABLE table=test-table [id=ef42e4cf00704b32b1f1f0c13eed898c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.577309 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f32fa567444ffa95c85ccf40b8f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.577333 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f32fa567444ffa95c85ccf40b8f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.578727 5479 tablet_bootstrap.cc:492] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Bootstrap starting.
I20260812 14:28:22.578764 5324 tablet_service.cc:1467] Processing CreateTablet for tablet 84f32fa567444ffa95c85ccf40b8f9b1 (DEFAULT_TABLE table=test-table [id=ef42e4cf00704b32b1f1f0c13eed898c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.578881 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f32fa567444ffa95c85ccf40b8f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.579083 5479 tablet_bootstrap.cc:654] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.579808 5485 tablet_bootstrap.cc:492] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Bootstrap starting.
I20260812 14:28:22.580089 5479 tablet_bootstrap.cc:492] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: No bootstrap required, opened a new log
I20260812 14:28:22.580132 5479 ts_tablet_manager.cc:1397] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:22.580246 5485 tablet_bootstrap.cc:654] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.580277 5479 raft_consensus.cc:348] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.580344 5479 raft_consensus.cc:374] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.580363 5479 raft_consensus.cc:729] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817e4cf002447318b6a6b0108744db9, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.580416 5479 consensus_queue.cc:260] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.580506 5479 ts_tablet_manager.cc:1428] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.581543 5478 tablet_bootstrap.cc:492] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Bootstrap starting.
I20260812 14:28:22.581943 5478 tablet_bootstrap.cc:654] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.582029 5485 tablet_bootstrap.cc:492] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: No bootstrap required, opened a new log
I20260812 14:28:22.582074 5485 ts_tablet_manager.cc:1397] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:28:22.582233 5485 raft_consensus.cc:348] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.582301 5485 raft_consensus.cc:374] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.582319 5485 raft_consensus.cc:729] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851d1ae37b1e4910bcc6be1dd9f4a8b0, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.582372 5485 consensus_queue.cc:260] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.582463 5485 ts_tablet_manager.cc:1428] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.582962 5478 tablet_bootstrap.cc:492] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: No bootstrap required, opened a new log
I20260812 14:28:22.583011 5478 ts_tablet_manager.cc:1397] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:22.583114 5478 raft_consensus.cc:348] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.583174 5478 raft_consensus.cc:374] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.583192 5478 raft_consensus.cc:729] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.583237 5478 consensus_queue.cc:260] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.583335 5478 ts_tablet_manager.cc:1428] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.648525 5507 raft_consensus.cc:482] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:28:22.648633 5507 raft_consensus.cc:504] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.648792 5507 leader_election.cc:288] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319), 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999)
I20260812 14:28:22.649039 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f32fa567444ffa95c85ccf40b8f9b1" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" is_pre_election: true
I20260812 14:28:22.649039 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f32fa567444ffa95c85ccf40b8f9b1" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9" is_pre_election: true
I20260812 14:28:22.649148 5140 raft_consensus.cc:2454] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 0.
I20260812 14:28:22.649148 4910 raft_consensus.cc:2454] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 0.
I20260812 14:28:22.649323 5249 leader_election.cc:302] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9817e4cf002447318b6a6b0108744db9, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.649430 5507 raft_consensus.cc:2781] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:28:22.649466 5507 raft_consensus.cc:482] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:28:22.649492 5507 raft_consensus.cc:3037] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.650244 5507 raft_consensus.cc:504] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.650362 5507 leader_election.cc:288] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 election: Requested vote from peers 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319), 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999)
I20260812 14:28:22.650568 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f32fa567444ffa95c85ccf40b8f9b1" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9"
I20260812 14:28:22.650556 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f32fa567444ffa95c85ccf40b8f9b1" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
I20260812 14:28:22.650641 4910 raft_consensus.cc:3037] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.650641 5140 raft_consensus.cc:3037] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.651386 4910 raft_consensus.cc:2454] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 1.
I20260812 14:28:22.651386 5140 raft_consensus.cc:2454] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 1.
I20260812 14:28:22.651553 5254 leader_election.cc:302] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.651679 5507 raft_consensus.cc:2781] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:22.651762 5507 raft_consensus.cc:686] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 1 LEADER]: Becoming Leader. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Running, Role: LEADER
I20260812 14:28:22.651814 5507 consensus_queue.cc:237] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.652407 4678 catalog_manager.cc:5495] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7e8ba71830846f581d50b39e1e68604 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "f7e8ba71830846f581d50b39e1e68604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } health_report { overall_health: HEALTHY } } }
I20260812 14:28:22.657852 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.655741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.23.173.195:34875)
0812 14:28:22.655791 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.655793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.655803 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:22.655850 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.655851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.655862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.655924 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.656058 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.656535 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.657248 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.657441 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.657497 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.657633 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.657668 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.657707 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.657761 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.657782 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:28:22.658603 5140 raft_consensus.cc:1261] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Refusing update from remote peer f7e8ba71830846f581d50b39e1e68604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.658604 4910 raft_consensus.cc:1261] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Refusing update from remote peer f7e8ba71830846f581d50b39e1e68604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.658825 5507 consensus_queue.cc:1035] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.658946 5486 consensus_queue.cc:1035] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.661547 4677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.661638 4677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.662169 4677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Sending DeleteTablet for 3 replicas of tablet 84f32fa567444ffa95c85ccf40b8f9b1
I20260812 14:28:22.662400 4878 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f32fa567444ffa95c85ccf40b8f9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 14:28:22.662444 5324 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f32fa567444ffa95c85ccf40b8f9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36908
I20260812 14:28:22.662496 5511 tablet_replica.cc:330] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: stopping tablet replica
I20260812 14:28:22.662541 5511 raft_consensus.cc:2229] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.662585 5511 raft_consensus.cc:2258] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.662562 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f32fa567444ffa95c85ccf40b8f9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:58964
I20260812 14:28:22.662654 5513 tablet_replica.cc:330] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: stopping tablet replica
I20260812 14:28:22.662739 5513 raft_consensus.cc:2229] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:28:22.662802 5513 raft_consensus.cc:2258] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.662757 4677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 14:28:22.662899 5512 tablet_replica.cc:330] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: stopping tablet replica
W20260812 14:28:22.662910 4677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:28:22.662951 5512 raft_consensus.cc:2229] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.663000 5512 raft_consensus.cc:2258] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.663321 5513 ts_tablet_manager.cc:1905] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.663316 5512 ts_tablet_manager.cc:1905] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.665017 5513 ts_tablet_manager.cc:1918] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:22.665055 5512 ts_tablet_manager.cc:1918] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:22.665081 5513 log.cc:1196] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/84f32fa567444ffa95c85ccf40b8f9b1
I20260812 14:28:22.665112 5512 log.cc:1196] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/84f32fa567444ffa95c85ccf40b8f9b1
I20260812 14:28:22.665123 5511 ts_tablet_manager.cc:1905] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.665287 5513 ts_tablet_manager.cc:1939] T 84f32fa567444ffa95c85ccf40b8f9b1 P f7e8ba71830846f581d50b39e1e68604: Deleting consensus metadata
I20260812 14:28:22.665352 5512 ts_tablet_manager.cc:1939] T 84f32fa567444ffa95c85ccf40b8f9b1 P 9817e4cf002447318b6a6b0108744db9: Deleting consensus metadata
I20260812 14:28:22.665676 4647 catalog_manager.cc:4841] TS 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999): tablet 84f32fa567444ffa95c85ccf40b8f9b1 (table test-table [id=ef42e4cf00704b32b1f1f0c13eed898c]) successfully deleted
I20260812 14:28:22.665697 4648 catalog_manager.cc:4841] TS f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875): tablet 84f32fa567444ffa95c85ccf40b8f9b1 (table test-table [id=ef42e4cf00704b32b1f1f0c13eed898c]) successfully deleted
I20260812 14:28:22.666463 5511 ts_tablet_manager.cc:1918] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:22.666525 5511 log.cc:1196] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/84f32fa567444ffa95c85ccf40b8f9b1
I20260812 14:28:22.666576 4878 tablet_service.cc:1467] Processing CreateTablet for tablet dca10b402ec1459cb1571b82605efd4e (DEFAULT_TABLE table=test-table [id=48718011ead249dba79e4e17f0efbaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.666610 5106 tablet_service.cc:1467] Processing CreateTablet for tablet dca10b402ec1459cb1571b82605efd4e (DEFAULT_TABLE table=test-table [id=48718011ead249dba79e4e17f0efbaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.666700 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca10b402ec1459cb1571b82605efd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.666708 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca10b402ec1459cb1571b82605efd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.666723 5511 ts_tablet_manager.cc:1939] T 84f32fa567444ffa95c85ccf40b8f9b1 P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Deleting consensus metadata
I20260812 14:28:22.667040 4650 catalog_manager.cc:4841] TS 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319): tablet 84f32fa567444ffa95c85ccf40b8f9b1 (table test-table [id=ef42e4cf00704b32b1f1f0c13eed898c]) successfully deleted
I20260812 14:28:22.667586 5485 tablet_bootstrap.cc:492] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Bootstrap starting.
I20260812 14:28:22.667918 5485 tablet_bootstrap.cc:654] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.667917 5479 tablet_bootstrap.cc:492] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Bootstrap starting.
I20260812 14:28:22.667960 5324 tablet_service.cc:1467] Processing CreateTablet for tablet dca10b402ec1459cb1571b82605efd4e (DEFAULT_TABLE table=test-table [id=48718011ead249dba79e4e17f0efbaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:22.668057 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca10b402ec1459cb1571b82605efd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:22.668205 5479 tablet_bootstrap.cc:654] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.668735 5479 tablet_bootstrap.cc:492] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: No bootstrap required, opened a new log
I20260812 14:28:22.668790 5479 ts_tablet_manager.cc:1397] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:22.668891 5479 raft_consensus.cc:348] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.668967 5479 raft_consensus.cc:374] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.668983 5479 raft_consensus.cc:729] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817e4cf002447318b6a6b0108744db9, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.669032 5479 consensus_queue.cc:260] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.669128 5479 ts_tablet_manager.cc:1428] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.669194 5478 tablet_bootstrap.cc:492] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Bootstrap starting.
I20260812 14:28:22.669471 5478 tablet_bootstrap.cc:654] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:22.669718 5485 tablet_bootstrap.cc:492] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0: No bootstrap required, opened a new log
I20260812 14:28:22.669770 5485 ts_tablet_manager.cc:1397] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:28:22.669883 5485 raft_consensus.cc:348] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.669956 5485 raft_consensus.cc:374] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.669974 5485 raft_consensus.cc:729] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851d1ae37b1e4910bcc6be1dd9f4a8b0, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.670015 5485 consensus_queue.cc:260] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.670064 5478 tablet_bootstrap.cc:492] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: No bootstrap required, opened a new log
I20260812 14:28:22.670105 5485 ts_tablet_manager.cc:1428] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.670112 5478 ts_tablet_manager.cc:1397] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:22.670212 5478 raft_consensus.cc:348] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.670280 5478 raft_consensus.cc:374] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:22.670301 5478 raft_consensus.cc:729] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Initialized, Role: FOLLOWER
I20260812 14:28:22.670356 5478 consensus_queue.cc:260] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.670449 5478 ts_tablet_manager.cc:1428] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:22.704954 5507 raft_consensus.cc:482] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:28:22.705070 5507 raft_consensus.cc:504] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.705225 5507 leader_election.cc:288] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319), 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999)
I20260812 14:28:22.705466 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca10b402ec1459cb1571b82605efd4e" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9" is_pre_election: true
I20260812 14:28:22.705467 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca10b402ec1459cb1571b82605efd4e" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" is_pre_election: true
I20260812 14:28:22.705566 5140 raft_consensus.cc:2454] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 0.
I20260812 14:28:22.705570 4910 raft_consensus.cc:2454] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 0.
I20260812 14:28:22.705721 5249 leader_election.cc:302] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9817e4cf002447318b6a6b0108744db9, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.705847 5507 raft_consensus.cc:2781] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:28:22.705904 5507 raft_consensus.cc:482] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:28:22.705925 5507 raft_consensus.cc:3037] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.706657 5507 raft_consensus.cc:504] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.706792 5507 leader_election.cc:288] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 election: Requested vote from peers 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319), 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999)
I20260812 14:28:22.706960 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca10b402ec1459cb1571b82605efd4e" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817e4cf002447318b6a6b0108744db9"
I20260812 14:28:22.706974 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca10b402ec1459cb1571b82605efd4e" candidate_uuid: "f7e8ba71830846f581d50b39e1e68604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0"
I20260812 14:28:22.707047 4910 raft_consensus.cc:3037] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.707048 5140 raft_consensus.cc:3037] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:22.707844 5140 raft_consensus.cc:2454] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 1.
I20260812 14:28:22.707844 4910 raft_consensus.cc:2454] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e8ba71830846f581d50b39e1e68604 in term 1.
I20260812 14:28:22.708033 5254 leader_election.cc:302] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851d1ae37b1e4910bcc6be1dd9f4a8b0, f7e8ba71830846f581d50b39e1e68604; no voters:
I20260812 14:28:22.708138 5507 raft_consensus.cc:2781] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:22.708192 5507 raft_consensus.cc:686] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 1 LEADER]: Becoming Leader. State: Replica: f7e8ba71830846f581d50b39e1e68604, State: Running, Role: LEADER
I20260812 14:28:22.708242 5507 consensus_queue.cc:237] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } }
I20260812 14:28:22.708778 4677 catalog_manager.cc:5495] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7e8ba71830846f581d50b39e1e68604 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "f7e8ba71830846f581d50b39e1e68604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7e8ba71830846f581d50b39e1e68604" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34875 } health_report { overall_health: HEALTHY } } }
I20260812 14:28:22.718371 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.716186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:45073 (local address 127.23.173.194:41999)
0812 14:28:22.716238 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.716240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.716305 (+ 65us) server_negotiation.cc:407] Connection header received
0812 14:28:22.716357 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.716358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.716366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.716414 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.716525 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.716991 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.717747 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.717954 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.718012 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.718170 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.718202 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.718249 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.718290 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.718313 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:28:22.718379 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.716124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:41999 (local address 127.23.173.195:45073)
0812 14:28:22.716254 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.716265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.716319 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.716421 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.716421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.716423 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:28:22.716482 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.716483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.717055 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.717055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.717651 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.717653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.718021 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.718021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.718065 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.718081 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.718253 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.718254 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.718279 (+ 25us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.718317 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260812 14:28:22.720337 4910 raft_consensus.cc:1261] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Refusing update from remote peer f7e8ba71830846f581d50b39e1e68604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.720335 5140 raft_consensus.cc:1261] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Refusing update from remote peer f7e8ba71830846f581d50b39e1e68604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:22.720587 5486 consensus_queue.cc:1035] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851d1ae37b1e4910bcc6be1dd9f4a8b0" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.720721 5507 consensus_queue.cc:1035] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9817e4cf002447318b6a6b0108744db9" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:22.726963 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.724886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:34085 (local address 127.23.173.195:34875)
0812 14:28:22.724923 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:22.724925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:22.724948 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:28:22.725009 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:22.725010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:22.725020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:22.725076 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:22.725182 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.725670 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.726352 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.726556 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.726615 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.726744 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.726782 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.726829 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:22.726877 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 14:28:22.726900 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:28:22.727016 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.724854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:34875 (local address 127.23.173.194:34085)
0812 14:28:22.724904 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.724915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:22.724963 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:22.725079 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:22.725080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:22.725081 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:28:22.725123 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.725125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.725738 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.725739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.726283 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:22.726285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:22.726615 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:22.726616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:22.726670 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:22.726677 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:28:22.726830 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:28:22.726832 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:28:22.726876 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:28:22.726931 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:28:22.732600 4678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.732683 4678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:22.733232 4678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6585b6461cca4a8b9e6b4c3f1d7d4dc6: Sending DeleteTablet for 3 replicas of tablet dca10b402ec1459cb1571b82605efd4e
I20260812 14:28:22.733451 4878 tablet_service.cc:1514] Processing DeleteTablet for tablet dca10b402ec1459cb1571b82605efd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 14:28:22.733566 5324 tablet_service.cc:1514] Processing DeleteTablet for tablet dca10b402ec1459cb1571b82605efd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36908
I20260812 14:28:22.733570 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet dca10b402ec1459cb1571b82605efd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:22 UTC) from {username='oryx', principal='oryx/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:58964
I20260812 14:28:22.733632 5511 tablet_replica.cc:330] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0: stopping tablet replica
I20260812 14:28:22.733645 5513 tablet_replica.cc:330] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: stopping tablet replica
I20260812 14:28:22.733691 5513 raft_consensus.cc:2229] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:28:22.733690 5511 raft_consensus.cc:2229] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.733731 5511 raft_consensus.cc:2258] T dca10b402ec1459cb1571b82605efd4e P 851d1ae37b1e4910bcc6be1dd9f4a8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.733745 5513 raft_consensus.cc:2258] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.733812 5512 tablet_replica.cc:330] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: stopping tablet replica
I20260812 14:28:22.733807 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 4783
I20260812 14:28:22.733876 5512 raft_consensus.cc:2229] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:22.733909 5512 raft_consensus.cc:2258] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:22.744675 5513 ts_tablet_manager.cc:1905] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.745908 5512 ts_tablet_manager.cc:1905] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:22.746214 5513 ts_tablet_manager.cc:1918] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:28:22.746273 5513 log.cc:1196] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/dca10b402ec1459cb1571b82605efd4e
I20260812 14:28:22.746443 5513 ts_tablet_manager.cc:1939] T dca10b402ec1459cb1571b82605efd4e P f7e8ba71830846f581d50b39e1e68604: Deleting consensus metadata
I20260812 14:28:22.746793 4648 catalog_manager.cc:4841] TS f7e8ba71830846f581d50b39e1e68604 (127.23.173.195:34875): tablet dca10b402ec1459cb1571b82605efd4e (table test-table [id=48718011ead249dba79e4e17f0efbaeb]) successfully deleted
I20260812 14:28:22.747186 5512 ts_tablet_manager.cc:1918] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:28:22.747238 5512 log.cc:1196] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/dca10b402ec1459cb1571b82605efd4e
I20260812 14:28:22.747408 5512 ts_tablet_manager.cc:1939] T dca10b402ec1459cb1571b82605efd4e P 9817e4cf002447318b6a6b0108744db9: Deleting consensus metadata
I20260812 14:28:22.747965 4647 catalog_manager.cc:4841] TS 9817e4cf002447318b6a6b0108744db9 (127.23.173.194:41999): tablet dca10b402ec1459cb1571b82605efd4e (table test-table [id=48718011ead249dba79e4e17f0efbaeb]) successfully deleted
W20260812 14:28:22.752970 4650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:33319 (error 108)
I20260812 14:28:22.753232 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 4996
I20260812 14:28:22.758522 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:22.753294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33319 (local address 127.0.0.1:48022)
0812 14:28:22.758407 (+ 5113us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:22.758475 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5089,"thread_start_us":39,"threads_started":1}
W20260812 14:28:22.758630 4650 catalog_manager.cc:4584] TS 851d1ae37b1e4910bcc6be1dd9f4a8b0 (127.23.173.193:33319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dca10b402ec1459cb1571b82605efd4e: Network error: Client connection negotiation failed: client connection to 127.23.173.193:33319: connect: Connection refused (error 111)
I20260812 14:28:22.769511 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 5226
I20260812 14:28:22.781587 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 4612
2026-08-12T14:28:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2643 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[5630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[5630](info): set up 2 sockets
Aug 12 14:28:22 dist-test-slave-1l3l krb5kdc[5630](info): commencing operation
krb5kdc: starting...
W20260812 14:28:23.961565 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:28:24 dist-test-slave-1l3l krb5kdc[5630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:28:24.083231 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43309
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:43309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:24.165980 5659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:24.166180 5659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:24.166203 5659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:24.167282 5659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:24.167315 5659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:24.167328 5659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:24.167339 5659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:24.167349 5659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:24.168831 5659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:43309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:24.169041 5659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:24.169219 5659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:24.173904 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:24.173949 5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:24.173904 5674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:24.174238 5659 server_base.cc:1034] running on GCE node
I20260812 14:28:24.174508 5659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:24.174796 5659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:24.175959 5659 hybrid_clock.cc:648] HybridClock initialized: now 1786544904175936 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T14:28:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1405 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:28:24 dist-test-slave-1l3l krb5kdc[5682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:28:24 dist-test-slave-1l3l krb5kdc[5682](info): set up 2 sockets
Aug 12 14:28:24 dist-test-slave-1l3l krb5kdc[5682](info): commencing operation
krb5kdc: starting...
W20260812 14:28:25.372996 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:28:25.494377 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46653
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:25.574424 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:25.574633 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:25.574663 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:25.575834 5701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:25.575875 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:25.575894 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:25.575909 5701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:25.575924 5701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:25.577400 5701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:25.577646 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:25.577857 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:25.582528 5709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:25.582608 5701 server_base.cc:1034] running on GCE node
W20260812 14:28:25.582535 5710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:25.582542 5712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:25.582996 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:25.583302 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:25.584448 5701 hybrid_clock.cc:648] HybridClock initialized: now 1786544905584426 us; error 41 us; skew 500 ppm
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:25.586850 5701 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:28:25.587260 5701 webserver.cc:460] Webserver started at http://127.23.173.254:42985/ using document root <none> and password file <none>
I20260812 14:28:25.587473 5701 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:25.587522 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:25.587646 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:25.588709 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "22df56d6c7e7401eac222f8b926bf3fb"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "cd32e23cdf4ecf0c9a86bded72fd0f8e"
server_key_iv: "3578bf79b7e044b9f56710438db0a2b4"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.589038 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "22df56d6c7e7401eac222f8b926bf3fb"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "cd32e23cdf4ecf0c9a86bded72fd0f8e"
server_key_iv: "3578bf79b7e044b9f56710438db0a2b4"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.590240 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:28:25.590878 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.591022 5701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:25.591109 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "22df56d6c7e7401eac222f8b926bf3fb"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "cd32e23cdf4ecf0c9a86bded72fd0f8e"
server_key_iv: "3578bf79b7e044b9f56710438db0a2b4"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.591167 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:25.610867 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:25.613070 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:25.613207 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:25.616947 5701 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46653
I20260812 14:28:25.616993 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46653 every 8 connection(s)
I20260812 14:28:25.617321 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:28:25.618147 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:25.618904 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 5701
I20260812 14:28:25.618996 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:28:25.619174 24247 external_mini_cluster.cc:1428] Setting key e718c816f564e526b0ac97c758d725a4
I20260812 14:28:25.621053 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Bootstrap starting.
I20260812 14:28:25.621752 5817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:25.622052 5817 log.cc:824] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:25.622709 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: No bootstrap required, opened a new log
I20260812 14:28:25.624531 5817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:25.624665 5817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:25.624687 5817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22df56d6c7e7401eac222f8b926bf3fb, State: Initialized, Role: FOLLOWER
I20260812 14:28:25.624795 5817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:25.624872 5817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:28:25.624907 5817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:28:25.624951 5817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:25.625622 5817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:25.625721 5817 leader_election.cc:302] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22df56d6c7e7401eac222f8b926bf3fb; no voters:
I20260812 14:28:25.625880 5817 leader_election.cc:288] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:28:25.625918 5823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:25.626044 5823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 LEADER]: Becoming Leader. State: Replica: 22df56d6c7e7401eac222f8b926bf3fb, State: Running, Role: LEADER
I20260812 14:28:25.626156 5817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:28:25.626200 5823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:25.626585 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22df56d6c7e7401eac222f8b926bf3fb. Latest consensus state: current_term: 1 leader_uuid: "22df56d6c7e7401eac222f8b926bf3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } } }
I20260812 14:28:25.626659 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:25.626667 5825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22df56d6c7e7401eac222f8b926bf3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } } }
I20260812 14:28:25.626760 5825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:25.627205 5829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:28:25.627688 5829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:28:25.629994 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:25.619641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.23.173.254:46653)
0812 14:28:25.619830 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:25.619839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:25.619874 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:28:25.620333 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:25.620342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:25.620409 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:25.620546 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:25.621133 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.622384 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.623160 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.623572 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.623667 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:25.625867 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:25.625870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:25.625870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:25.625881 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:25.628014 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:25.628426 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:25.628429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:25.628429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:25.628535 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:25.628755 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:25.628756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:25.628756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:25.629059 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:25.629573 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:25.629646 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 14:28:25.629782 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260812 14:28:25.632136 5829 catalog_manager.cc:1347] Generated new cluster ID: 2a1caeadc25945689481b08aff4b6132
I20260812 14:28:25.632194 5829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:28:25.639477 5829 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:28:25.640064 5829 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:28:25.644444 5829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Generated new TSK 0
I20260812 14:28:25.644618 5829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:28:25.670217 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46653
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:25.752712 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:25.752982 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:25.753021 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:25.754545 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:25.754592 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:25.754628 5848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:28:25.756161 5848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:25.756415 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:25.756636 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:25.761758 5856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:25.761758 5859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:25.761785 5855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:25.761874 5848 server_base.cc:1034] running on GCE node
I20260812 14:28:25.762178 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:25.762485 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:25.763726 5848 hybrid_clock.cc:648] HybridClock initialized: now 1786544905763683 us; error 29 us; skew 500 ppm
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:25.766259 5848 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:28:25.766765 5848 webserver.cc:460] Webserver started at http://127.23.173.193:42987/ using document root <none> and password file <none>
I20260812 14:28:25.766992 5848 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:25.767043 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:25.767171 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:25.768311 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "5695b7e2af2c42138bcdd6dc41ea0e28"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "60dcbeae1029acdd2202e4f780877f13"
server_key_iv: "f4f39019988a9040f4c03fa7b887ea5f"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.768668 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "5695b7e2af2c42138bcdd6dc41ea0e28"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "60dcbeae1029acdd2202e4f780877f13"
server_key_iv: "f4f39019988a9040f4c03fa7b887ea5f"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.769969 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:28:25.770696 5870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.770864 5848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:25.770967 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "5695b7e2af2c42138bcdd6dc41ea0e28"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "60dcbeae1029acdd2202e4f780877f13"
server_key_iv: "f4f39019988a9040f4c03fa7b887ea5f"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.771027 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:25.780926 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:25.782732 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:25.782871 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:25.783094 5848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:25.783407 5848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:25.783445 5848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.783475 5848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:25.783501 5848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.789431 5848 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:34731
I20260812 14:28:25.789500 6042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:34731 every 8 connection(s)
I20260812 14:28:25.789801 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:28:25.795558 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 5848
I20260812 14:28:25.795650 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:25.795879 24247 external_mini_cluster.cc:1428] Setting key 4af694843a0386f70828ceddaaad5539
I20260812 14:28:25.798784 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:25.790509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:59405 (local address 127.23.173.254:46653)
0812 14:28:25.790555 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:25.790557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:25.790766 (+ 209us) server_negotiation.cc:407] Connection header received
0812 14:28:25.791241 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:25.791242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:25.791251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:25.791308 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:25.792311 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.792818 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.793810 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.794021 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.794073 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:25.796174 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:25.796175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:25.796175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:25.796182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:25.797503 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:25.797880 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:25.797882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:25.797882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:25.797942 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:25.798164 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:25.798165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:25.798165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:25.798281 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:25.798337 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:25.798600 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 14:28:25.798693 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:28:25.798941 6046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:25.790470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:59405)
0812 14:28:25.790643 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:25.790686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:25.791165 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:25.791376 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:25.791379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:25.792026 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:25.792227 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:25.792233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.792883 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.792884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:25.793745 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:25.793748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.794081 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.794081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:25.794176 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:25.794612 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:25.794620 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:25.796074 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:25.797581 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:25.797590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:25.797592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:25.797811 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:25.797999 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:25.798000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:25.798000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:25.798108 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:25.798337 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:25.798345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:25.798482 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 14:28:25.798571 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:28:25.798727 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260812 14:28:25.799604 6044 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46653
I20260812 14:28:25.799690 6044 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:25.799887 6044 heartbeater.cc:511] Master 127.23.173.254:46653 requested a full tablet report, sending...
I20260812 14:28:25.800438 5751 ts_manager.cc:194] Registered new tserver with Master: 5695b7e2af2c42138bcdd6dc41ea0e28 (127.23.173.193:34731)
I20260812 14:28:25.801386 5751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:59405
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:28:25.822813 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46653
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:25.904739 6051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:25.904949 6051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:25.904973 6051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:25.906154 6051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:25.906200 6051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:25.906227 6051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:28:25.907667 6051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:25.907913 6051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:25.908142 6051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:25.913503 6062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:25.913580 6051 server_base.cc:1034] running on GCE node
W20260812 14:28:25.913589 6066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:25.913503 6063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:25.913954 6051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:25.914196 6051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:25.915347 6051 hybrid_clock.cc:648] HybridClock initialized: now 1786544905915322 us; error 32 us; skew 500 ppm
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:25.917771 6051 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:28:25.918207 6051 webserver.cc:460] Webserver started at http://127.23.173.194:35309/ using document root <none> and password file <none>
I20260812 14:28:25.918388 6051 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:25.918427 6051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:25.918530 6051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:25.919565 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "23ac782f57e54aa699eb525b916c66cc"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "9fd3558cb5adb09f9bbf5d7ec8887538"
server_key_iv: "cbb229c3d22e2cd4ed80b2a0a75de7d5"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.919924 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "23ac782f57e54aa699eb525b916c66cc"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "9fd3558cb5adb09f9bbf5d7ec8887538"
server_key_iv: "cbb229c3d22e2cd4ed80b2a0a75de7d5"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.921165 6051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:25.921810 6079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.921958 6051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.922041 6051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "23ac782f57e54aa699eb525b916c66cc"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "9fd3558cb5adb09f9bbf5d7ec8887538"
server_key_iv: "cbb229c3d22e2cd4ed80b2a0a75de7d5"
server_key_version: "encryptionkey@0"
I20260812 14:28:25.922101 6051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:25.935575 6051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:25.937294 6051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:25.937433 6051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:25.937655 6051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:25.937958 6051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:25.937994 6051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.938021 6051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:25.938038 6051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:25.943486 6051 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:35733
I20260812 14:28:25.943526 6260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:35733 every 8 connection(s)
I20260812 14:28:25.943854 6051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:28:25.948423 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 6051
I20260812 14:28:25.948552 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:28:25.948755 24247 external_mini_cluster.cc:1428] Setting key b5f97fa69f879ab5b1957754e2a25f12
Aug 12 14:28:25 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544905, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:25.953359 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:25.944509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60083 (local address 127.23.173.254:46653)
0812 14:28:25.944576 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:25.944578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:25.944754 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:28:25.945279 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:25.945280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:25.945292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:25.945341 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:25.946291 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.946796 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.947778 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.948036 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.948096 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:25.950492 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:25.950493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:25.950494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:25.950501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:25.952081 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:25.952462 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:25.952463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:25.952464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:25.952527 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:25.952735 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:25.952736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:25.952736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:25.952867 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:25.952931 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:25.953191 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 14:28:25.953273 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:28:25.953496 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:25.944465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:60083)
0812 14:28:25.944635 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:25.944676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:25.945178 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:25.945425 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:25.945430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:25.945990 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:25.946192 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:25.946198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.946880 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.946881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:25.947678 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:25.947681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:25.948113 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:25.948114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:25.948240 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:25.948751 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:25.948759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:25.950371 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:25.952158 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:25.952164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:25.952166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:25.952382 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:25.952592 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:25.952593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:25.952593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:25.952682 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:25.952923 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:25.952926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:25.953035 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 14:28:25.953161 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 14:28:25.953311 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260812 14:28:25.954007 6261 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46653
I20260812 14:28:25.954123 6261 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:25.954300 6261 heartbeater.cc:511] Master 127.23.173.254:46653 requested a full tablet report, sending...
I20260812 14:28:25.954650 5751 ts_manager.cc:194] Registered new tserver with Master: 23ac782f57e54aa699eb525b916c66cc (127.23.173.194:35733)
I20260812 14:28:25.954982 5751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:60083
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:28:25.974327 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46653
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:26.058737 6270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:26.058983 6270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:26.059021 6270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:26.060286 6270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:26.060338 6270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:26.060374 6270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:28:26.061892 6270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:26.062167 6270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:26.062397 6270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:26.067533 6276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.067656 6270 server_base.cc:1034] running on GCE node
W20260812 14:28:26.067603 6277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:26.067543 6279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.068007 6270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:26.068291 6270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:26.069485 6270 hybrid_clock.cc:648] HybridClock initialized: now 1786544906069469 us; error 42 us; skew 500 ppm
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544906, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:26.072381 6270 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:28:26.074510 6270 webserver.cc:460] Webserver started at http://127.23.173.195:38777/ using document root <none> and password file <none>
I20260812 14:28:26.074751 6270 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:26.074803 6270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:26.074946 6270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:26.076131 6270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "00beb8dac70447b2b787a877421b94b0"
format_stamp: "Formatted at 2026-08-12 14:28:26 on dist-test-slave-1l3l"
server_key: "bd20d09ecb7b97453506e401bdb7b2ef"
server_key_iv: "4cdb568981bffe741f6eb9ca690ea12e"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.076485 6270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "00beb8dac70447b2b787a877421b94b0"
format_stamp: "Formatted at 2026-08-12 14:28:26 on dist-test-slave-1l3l"
server_key: "bd20d09ecb7b97453506e401bdb7b2ef"
server_key_iv: "4cdb568981bffe741f6eb9ca690ea12e"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.077836 6270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:28:26.078711 6292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.078928 6270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:26.079044 6270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "00beb8dac70447b2b787a877421b94b0"
format_stamp: "Formatted at 2026-08-12 14:28:26 on dist-test-slave-1l3l"
server_key: "bd20d09ecb7b97453506e401bdb7b2ef"
server_key_iv: "4cdb568981bffe741f6eb9ca690ea12e"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.079110 6270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:26.096206 6270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:26.098527 6270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:26.098683 6270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:26.098937 6270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:26.099257 6270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:26.099294 6270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.099325 6270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:26.099359 6270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.106081 6270 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:36705
I20260812 14:28:26.106216 6480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:36705 every 8 connection(s)
I20260812 14:28:26.106508 6270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:28:26.109673 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 6270
I20260812 14:28:26.109787 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:28:26.109974 24247 external_mini_cluster.cc:1428] Setting key 970afab4e151bd6f1f2cce2b979d98c5
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544906, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:26.117267 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.107352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:52865 (local address 127.23.173.254:46653)
0812 14:28:26.107426 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.107429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.107638 (+ 209us) server_negotiation.cc:407] Connection header received
0812 14:28:26.108247 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.108249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.108264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.108317 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.109329 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.109983 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.111262 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.111514 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.111578 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.114341 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:26.114342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:26.114343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:26.114350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:26.115998 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:26.116429 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:26.116429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:26.116430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:26.116481 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:26.116697 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:26.116697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:26.116697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:26.116810 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:26.116857 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:26.117129 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 14:28:26.117199 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:28:26.117481 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.107316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:52865)
0812 14:28:26.107500 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.107549 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.108139 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.108396 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.108401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.109011 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.109226 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.109230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.110120 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.110121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.111180 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.111184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.111622 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.111623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.111789 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.112544 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.112555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.114226 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:26.116101 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:26.116110 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:26.116113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:26.116367 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:26.116529 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:26.116530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:26.116530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:26.116651 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:26.116862 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:26.116868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:26.117019 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 14:28:26.117115 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 14:28:26.117260 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260812 14:28:26.118048 6481 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46653
I20260812 14:28:26.118158 6481 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:26.118355 6481 heartbeater.cc:511] Master 127.23.173.254:46653 requested a full tablet report, sending...
I20260812 14:28:26.118793 5751 ts_manager.cc:194] Registered new tserver with Master: 00beb8dac70447b2b787a877421b94b0 (127.23.173.195:36705)
I20260812 14:28:26.119180 5751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:52865
I20260812 14:28:26.122190 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab.
Entry for principal oryx/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab.
WARNING: no policy specified for oryx/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab.
Entry for principal oryx/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab.
WARNING: no policy specified for oryx/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab.
Entry for principal oryx/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab.
I20260812 14:28:26.202315 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 5848
I20260812 14:28:26.215238 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 6051
I20260812 14:28:26.237154 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 6270
I20260812 14:28:26.280117 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 5701
I20260812 14:28:26.289773 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46653
--webserver_interface=127.23.173.254
--webserver_port=42985
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:26.372177 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:26.372385 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:26.372416 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:26.373579 6493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:26.373626 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:26.373644 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:26.373660 6493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:26.373677 6493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:26.375164 6493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:26.375428 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:26.375655 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:26.380566 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.380595 6493 server_base.cc:1034] running on GCE node
W20260812 14:28:26.380553 6505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:26.380539 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.381042 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:26.381356 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:26.382509 6493 hybrid_clock.cc:648] HybridClock initialized: now 1786544906382466 us; error 39 us; skew 500 ppm
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544906, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:26.385051 6493 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:28:26.385479 6493 webserver.cc:460] Webserver started at http://127.23.173.254:42985/ using document root <none> and password file <none>
I20260812 14:28:26.385707 6493 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:26.385777 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:26.387142 6493 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:26.387812 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.387960 6493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.388047 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "22df56d6c7e7401eac222f8b926bf3fb"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "cd32e23cdf4ecf0c9a86bded72fd0f8e"
server_key_iv: "3578bf79b7e044b9f56710438db0a2b4"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.388381 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:26.402302 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:26.404268 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:26.404415 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:26.408377 6493 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46653
I20260812 14:28:26.408419 6604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46653 every 8 connection(s)
I20260812 14:28:26.408792 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:28:26.410286 6610 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:28:26.410926 6610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Bootstrap starting.
I20260812 14:28:26.414894 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 6493
I20260812 14:28:26.415185 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:34731
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:46653
--webserver_port=42987
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:28:26.419781 6610 log.cc:824] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:26.421442 6610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:28:26.421723 6610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Bootstrap complete.
I20260812 14:28:26.423933 6610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:26.424177 6610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22df56d6c7e7401eac222f8b926bf3fb, State: Initialized, Role: FOLLOWER
I20260812 14:28:26.424325 6610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:26.424391 6610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:28:26.424437 6610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:28:26.424486 6610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:28:26.425475 6610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:26.425570 6610 leader_election.cc:302] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22df56d6c7e7401eac222f8b926bf3fb; no voters:
I20260812 14:28:26.425750 6610 leader_election.cc:288] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:28:26.425797 6619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:28:26.425957 6619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [term 2 LEADER]: Becoming Leader. State: Replica: 22df56d6c7e7401eac222f8b926bf3fb, State: Running, Role: LEADER
I20260812 14:28:26.426021 6610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:28:26.426139 6619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } }
I20260812 14:28:26.426783 6620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22df56d6c7e7401eac222f8b926bf3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } } }
I20260812 14:28:26.426966 6620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:26.426801 6622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22df56d6c7e7401eac222f8b926bf3fb. Latest consensus state: current_term: 2 leader_uuid: "22df56d6c7e7401eac222f8b926bf3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df56d6c7e7401eac222f8b926bf3fb" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46653 } } }
I20260812 14:28:26.427161 6634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:28:26.427177 6622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:26.427767 6634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:28:26.427973 6634 catalog_manager.cc:1259] Loaded cluster ID: 2a1caeadc25945689481b08aff4b6132
I20260812 14:28:26.428150 6634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:28:26.429389 6634 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:28:26.429688 6634 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22df56d6c7e7401eac222f8b926bf3fb: Loaded TSK: 0
I20260812 14:28:26.429939 6634 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:28:26.499493 6616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:26.499725 6616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:26.499761 6616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:26.500908 6616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:26.500949 6616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:26.500984 6616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:28:26.502408 6616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.193.keytab
--principal=oryx/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:34731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=42987
--tserver_master_addrs=127.23.173.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:26.502645 6616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:26.502857 6616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:26.507921 6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.507966 6616 server_base.cc:1034] running on GCE node
W20260812 14:28:26.507934 6654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:26.508055 6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.508396 6616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:26.508632 6616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:26.509770 6616 hybrid_clock.cc:648] HybridClock initialized: now 1786544906509749 us; error 38 us; skew 500 ppm
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544906, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:26.512311 6616 init.cc:377] Logged in from keytab as oryx/127.23.173.193@KRBTEST.COM (short username oryx)
I20260812 14:28:26.512748 6616 webserver.cc:460] Webserver started at http://127.23.173.193:42987/ using document root <none> and password file <none>
I20260812 14:28:26.512993 6616 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:26.513055 6616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:26.514432 6616 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:26.515162 6662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.515333 6616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:26.515426 6616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "5695b7e2af2c42138bcdd6dc41ea0e28"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "60dcbeae1029acdd2202e4f780877f13"
server_key_iv: "f4f39019988a9040f4c03fa7b887ea5f"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.515812 6616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:26.525728 6616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:26.527915 6616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:26.528040 6616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:26.528257 6616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:26.528545 6616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:26.528579 6616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.528607 6616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:26.528632 6616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.534422 6616 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:34731
I20260812 14:28:26.534482 6866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:34731 every 8 connection(s)
I20260812 14:28:26.534829 6616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:28:26.541435 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 6616
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.541806 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:35733
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:46653
--webserver_port=35309
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.543239 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.535552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:50559)
0812 14:28:26.535736 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.535777 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.536270 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.536689 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.536693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.537262 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.537454 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.537459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.538551 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.538552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.539327 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.539330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.540202 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.540204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.540313 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.540828 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.540836 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.543040 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
W20260812 14:28:26.543329 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.535636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:50559 (local address 127.23.173.254:46653)
0812 14:28:26.535853 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.535861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.535909 (+ 48us) server_negotiation.cc:407] Connection header received
0812 14:28:26.536470 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.536474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.536518 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.536632 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.537572 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.538454 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.539434 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.540106 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.540229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.543181 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:50559: BlockingRecv error: recv got EOF from 127.23.173.193:50559 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":125,"threads_started":1}
W20260812 14:28:26.543726 6867 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:46653 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.173.254:46653: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.547791 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.543942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:54351)
0812 14:28:26.543988 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.543999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.544036 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.544153 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.544154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.544276 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.544320 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.544322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.544905 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.544906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.545397 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.545398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.545707 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.545708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.545770 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.546141 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.546146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.547691 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 14:28:26.547851 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.543992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:54351 (local address 127.23.173.254:46653)
0812 14:28:26.544024 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.544026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.544033 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:28:26.544084 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.544084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.544096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.544149 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.544384 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.544838 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.545466 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.545651 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.545705 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.547802 (+ 2097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:54351: BlockingRecv error: recv got EOF from 127.23.173.193:54351 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.552030 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.548226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:40577)
0812 14:28:26.548262 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.548271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.548303 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.548416 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.548417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.548541 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.548567 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.548568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.549131 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.549132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.549662 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.549664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.549964 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.549965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.550012 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.550356 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.550360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.551972 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:28:26.552114 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.548285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:40577 (local address 127.23.173.254:46653)
0812 14:28:26.548326 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.548328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.548335 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:28:26.548354 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.548355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.548363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.548413 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.548628 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.549065 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.549739 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.549911 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.549968 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.552061 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:40577: BlockingRecv error: recv got EOF from 127.23.173.193:40577 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:28:26.552282 6867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:28:26.625692 6877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:26.625903 6877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:26.625936 6877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:26.627156 6877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:26.627198 6877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:26.627234 6877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:28:26.628726 6877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.194.keytab
--principal=oryx/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=35309
--tserver_master_addrs=127.23.173.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:26.628958 6877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:26.629184 6877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:26.634008 6886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:26.634037 6888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.634258 6877 server_base.cc:1034] running on GCE node
W20260812 14:28:26.634367 6885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.634586 6877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:26.634827 6877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:26.635969 6877 hybrid_clock.cc:648] HybridClock initialized: now 1786544906635954 us; error 31 us; skew 500 ppm
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544906, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:26.638347 6877 init.cc:377] Logged in from keytab as oryx/127.23.173.194@KRBTEST.COM (short username oryx)
I20260812 14:28:26.638767 6877 webserver.cc:460] Webserver started at http://127.23.173.194:35309/ using document root <none> and password file <none>
I20260812 14:28:26.638974 6877 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:26.639036 6877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:26.640398 6877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:26.640936 6898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.641086 6877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:26.641170 6877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "23ac782f57e54aa699eb525b916c66cc"
format_stamp: "Formatted at 2026-08-12 14:28:25 on dist-test-slave-1l3l"
server_key: "9fd3558cb5adb09f9bbf5d7ec8887538"
server_key_iv: "cbb229c3d22e2cd4ed80b2a0a75de7d5"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.641503 6877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:26.651561 6877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:26.653676 6877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:26.653821 6877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:26.654047 6877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:26.654345 6877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:26.654381 6877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.654412 6877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:26.654436 6877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.660247 6877 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:35733
I20260812 14:28:26.660291 7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:35733 every 8 connection(s)
I20260812 14:28:26.660624 6877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:28:26.667646 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.661365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:44349 (local address 127.23.173.254:46653)
0812 14:28:26.661427 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.661429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.661621 (+ 192us) server_negotiation.cc:407] Connection header received
0812 14:28:26.662133 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.662134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.662151 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.662203 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.663121 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.663609 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.664611 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.664798 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.664853 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.667598 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:44349: BlockingRecv error: recv got EOF from 127.23.173.194:44349 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:28:26.667655 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.661348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:44349)
0812 14:28:26.661501 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.661537 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.662025 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.662270 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.662274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.662831 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.663021 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.663026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.663691 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.663692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.664522 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.664525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.664868 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.664869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.664986 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.665465 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.665474 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.667507 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260812 14:28:26.667922 7080 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:46653 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.173.254:46653: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
I20260812 14:28:26.668090 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 6877
I20260812 14:28:26.668412 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:36705
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:46653
--webserver_port=38777
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.672645 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.668129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:44811)
0812 14:28:26.668280 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.668293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.668350 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.668473 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.668474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.668606 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.668659 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.668661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.669465 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.669466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.670134 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.670138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.670567 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.670568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.670618 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.671009 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.671012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.672582 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260812 14:28:26.672724 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.668199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:44811 (local address 127.23.173.254:46653)
0812 14:28:26.668248 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.668249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.668335 (+ 86us) server_negotiation.cc:407] Connection header received
0812 14:28:26.668406 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.668407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.668415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.668456 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.668752 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.669358 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.670224 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.670494 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.670570 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.672670 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:44811: BlockingRecv error: recv got EOF from 127.23.173.194:44811 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.676697 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.672873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:34513)
0812 14:28:26.673015 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.673026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.673089 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.673205 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.673205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.673319 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.673345 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.673346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.673915 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.673916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.674399 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.674400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.674702 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.674703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.674752 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.675109 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.675114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.676633 (+ 1519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
W20260812 14:28:26.676815 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.672990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:34513 (local address 127.23.173.254:46653)
0812 14:28:26.673032 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.673034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.673066 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:28:26.673141 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.673142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.673151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.673196 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.673404 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.673850 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.674464 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.674647 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.674699 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.676712 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:34513: BlockingRecv error: recv got EOF from 127.23.173.194:34513 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:28:26.676819 7080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:28:26.753247 7085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:26.753454 7085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:26.753476 7085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:26.754607 7085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:26.754650 7085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:26.754678 7085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:28:26.756291 7085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/krb5kdc/oryx_127.23.173.195.keytab
--principal=oryx/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:36705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=38777
--tserver_master_addrs=127.23.173.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:26.756541 7085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:26.756752 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:26.761632 7096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:26.761668 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:26.761602 7098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:26.761813 7085 server_base.cc:1034] running on GCE node
I20260812 14:28:26.762025 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:26.762274 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:26.763412 7085 hybrid_clock.cc:648] HybridClock initialized: now 1786544906763399 us; error 35 us; skew 500 ppm
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544906, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:26.765874 7085 init.cc:377] Logged in from keytab as oryx/127.23.173.195@KRBTEST.COM (short username oryx)
I20260812 14:28:26.766278 7085 webserver.cc:460] Webserver started at http://127.23.173.195:38777/ using document root <none> and password file <none>
I20260812 14:28:26.766474 7085 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:26.766526 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:26.767823 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:26.768370 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.768537 7085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.768621 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "00beb8dac70447b2b787a877421b94b0"
format_stamp: "Formatted at 2026-08-12 14:28:26 on dist-test-slave-1l3l"
server_key: "bd20d09ecb7b97453506e401bdb7b2ef"
server_key_iv: "4cdb568981bffe741f6eb9ca690ea12e"
server_key_version: "encryptionkey@0"
I20260812 14:28:26.768978 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:26.783753 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:26.785650 7085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:26.785770 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:26.785967 7085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:26.786283 7085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:26.786314 7085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.786336 7085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:26.786352 7085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:26.792097 7085 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:36705
I20260812 14:28:26.792140 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:36705 every 8 connection(s)
I20260812 14:28:26.792507 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:28:26.794807 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 7085
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:28:26.800034 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.793197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:33853 (local address 127.23.173.254:46653)
0812 14:28:26.793263 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.793266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.793479 (+ 213us) server_negotiation.cc:407] Connection header received
0812 14:28:26.793953 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.793954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.793964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.794010 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.795003 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.795681 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.796665 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.796862 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.796918 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.799976 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:33853: BlockingRecv error: recv got EOF from 127.23.173.195:33853 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:28:26.800056 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.793190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:33853)
0812 14:28:26.793363 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.793400 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.793877 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.794073 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.794077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.794656 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.794862 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.794867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.795786 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.795787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.796583 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.796585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.796933 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.796934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.797061 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.797541 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.797550 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.799900 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
W20260812 14:28:26.800256 7302 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:46653 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.173.254:46653: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
I20260812 14:28:26.801967 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.795306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.23.173.254:46653)
0812 14:28:26.795459 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.795461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.795470 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:26.795498 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.795499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.795511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.795579 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.796163 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.796644 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.797362 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.797553 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.797606 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.798865 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:26.798870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:26.798871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:26.798882 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:26.800762 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:26.801065 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:26.801068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:26.801068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:26.801161 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:26.801326 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:26.801327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:26.801328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:26.801539 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:26.801660 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:26.801777 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 14:28:26.801878 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.804210 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.800456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:53831)
0812 14:28:26.800499 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.800509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.800546 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.800664 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.800664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.800769 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.800811 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.800812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.801376 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.801377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.801847 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.801849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.802219 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.802220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.802289 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.802619 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.802624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.804167 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:28:26.804301 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.800502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:53831 (local address 127.23.173.254:46653)
0812 14:28:26.800533 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.800535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.800552 (+ 17us) server_negotiation.cc:407] Connection header received
0812 14:28:26.800593 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.800594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.800602 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.800661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.800881 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.801311 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.801915 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.802145 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.802204 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.804243 (+ 2039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:53831: BlockingRecv error: recv got EOF from 127.23.173.195:53831 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:26 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:26.807515 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:26.804436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:41661)
0812 14:28:26.804464 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:26.804474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:26.804509 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:26.804609 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:26.804610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:26.804704 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:26.804729 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.804730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.805239 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.805239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.805695 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:26.805696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.805948 (+ 252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.805948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:26.805978 (+ 30us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.806254 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:26.806257 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:26.807476 (+ 1219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13}
W20260812 14:28:26.807616 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:26.804490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:41661 (local address 127.23.173.254:46653)
0812 14:28:26.804528 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:26.804530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:26.804537 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:28:26.804555 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:26.804556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:26.804564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:26.804605 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:26.804772 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.805188 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.805744 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:26.805904 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:26.805944 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:26.807561 (+ 1617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:41661: BlockingRecv error: recv got EOF from 127.23.173.195:41661 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:28:26.807641 7302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 14:28:27 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:27 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:27.557199 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:27.552657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:54899)
0812 14:28:27.552798 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:27.552809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:27.552870 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:27.553013 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:27.553014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:27.553183 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:27.553232 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:27.553234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.553876 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.553877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:27.554434 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:27.554436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.554814 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.554815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:27.554855 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:27.555303 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:27.555307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:27.557102 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260812 14:28:27.557281 7312 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:27.552726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:54899 (local address 127.23.173.254:46653)
0812 14:28:27.552847 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:27.552849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:27.552857 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:28:27.552930 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:27.552931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:27.552940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:27.552990 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:27.553288 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.553788 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.554541 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.554747 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.554810 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:27.557204 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:54899: BlockingRecv error: recv got EOF from 127.23.173.193:54899 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
Aug 12 14:28:27 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:27 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:27.681829 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:27.677175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:52815)
0812 14:28:27.677303 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:27.677313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:27.677369 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:27.677494 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:27.677496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:27.677661 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:27.677718 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:27.677719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.678337 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.678338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:27.678928 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:27.678930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.679293 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.679294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:27.679330 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:27.679850 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:27.679855 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:27.681739 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
W20260812 14:28:27.681879 7312 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:27.677264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:52815 (local address 127.23.173.254:46653)
0812 14:28:27.677322 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:27.677324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:27.677360 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:28:27.677420 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:27.677421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:27.677430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:27.677483 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:27.677775 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.678280 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.679025 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.679227 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.679285 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:27.681832 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:52815: BlockingRecv error: recv got EOF from 127.23.173.194:52815 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:28:27 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:27 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:27.813201 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:27.808150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:40181)
0812 14:28:27.808332 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:27.808344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:27.808422 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:27.808628 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:27.808629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:27.808793 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:27.808847 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:27.808848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.809495 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.809497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:27.810109 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:27.810111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.810485 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.810486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:27.810531 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:27.811074 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:27.811079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:27.813128 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
W20260812 14:28:27.813407 7312 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:27.808263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:40181 (local address 127.23.173.254:46653)
0812 14:28:27.808334 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:27.808337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:27.808491 (+ 154us) server_negotiation.cc:407] Connection header received
0812 14:28:27.808536 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:27.808538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:27.808554 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:27.808614 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:27.808898 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.809400 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.810206 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:27.810415 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:27.810472 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:27.813340 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:40181: BlockingRecv error: recv got EOF from 127.23.173.195:40181 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 14:28:28 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:28 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:28.562458 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:28.557774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:35619)
0812 14:28:28.557944 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:28.557959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:28.558024 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:28.558155 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:28.558155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:28.558319 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:28.558375 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:28.558377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.559249 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.559250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:28.559917 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:28.559920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.560238 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.560239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:28.560275 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:28.560679 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:28.560683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:28.562397 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
W20260812 14:28:28.562556 7320 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:28.557871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:35619 (local address 127.23.173.254:46653)
0812 14:28:28.558014 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:28.558016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:28.558025 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:28.558073 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:28.558074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:28.558085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:28.558141 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:28.558445 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.559179 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.559986 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.560182 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.560235 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:28.562492 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:35619: BlockingRecv error: recv got EOF from 127.23.173.193:35619 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
Aug 12 14:28:28 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:28 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:28.686741 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:28.682312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:58277)
0812 14:28:28.682457 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:28.682469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:28.682529 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:28.682661 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:28.682662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:28.682825 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:28.682876 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:28.682878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.683501 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.683502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:28.684072 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:28.684074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.684366 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.684367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:28.684403 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:28.684839 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:28.684843 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:28.686654 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260812 14:28:28.686844 7320 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:28.682445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:58277 (local address 127.23.173.254:46653)
0812 14:28:28.682503 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:28.682504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:28.682517 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:28:28.682587 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:28.682587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:28.682598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:28.682649 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:28.682930 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.683442 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.684134 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.684316 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.684363 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:28.686783 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:58277: BlockingRecv error: recv got EOF from 127.23.173.194:58277 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:28:28 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:28 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:28.818403 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:28.813808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:56767)
0812 14:28:28.813979 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:28.813991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:28.814053 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:28.814240 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:28.814241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:28.814410 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:28.814462 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:28.814464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.815130 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.815132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:28.815677 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:28.815679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.816100 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.816101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:28.816152 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:28.816581 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:28.816586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:28.818322 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20260812 14:28:28.818475 7320 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:28.813984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:56767 (local address 127.23.173.254:46653)
0812 14:28:28.814079 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:28.814081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:28.814091 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:28.814117 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:28.814118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:28.814131 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:28.814188 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:28.814543 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.815053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.815797 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:28.816031 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:28.816083 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:28.818419 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:56767: BlockingRecv error: recv got EOF from 127.23.173.195:56767 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 14:28:29 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:29 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:29.571431 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:29.562928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:53373)
0812 14:28:29.563091 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:29.563102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:29.563165 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:29.563288 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:29.563290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:29.563451 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:29.563503 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:29.563504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.564174 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.564175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:29.564709 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:29.564711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.565119 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.565120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:29.565187 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:29.569526 (+ 4339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:29.569532 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:29.571355 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
W20260812 14:28:29.571537 7325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:29.563038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:53373 (local address 127.23.173.254:46653)
0812 14:28:29.563162 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:29.563163 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:29.563173 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:29.563214 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:29.563214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:29.563225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:29.563273 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:29.563576 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.564102 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.564805 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.565039 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.565106 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:29.571456 (+ 6350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:53373: BlockingRecv error: recv got EOF from 127.23.173.193:53373 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
Aug 12 14:28:29 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:29 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:29.692212 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:29.687337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:57569)
0812 14:28:29.687577 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:29.687590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:29.687673 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:29.687853 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:29.687854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:29.688049 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:29.688100 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:29.688102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.688784 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.688785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:29.689315 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:29.689317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.689681 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.689683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:29.689743 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:29.690224 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:29.690229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:29.692134 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
W20260812 14:28:29.692301 7325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:29.687458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:57569 (local address 127.23.173.254:46653)
0812 14:28:29.687567 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:29.687569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:29.687647 (+ 78us) server_negotiation.cc:407] Connection header received
0812 14:28:29.687774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:29.687774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:29.687790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:29.687841 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:29.688180 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.688705 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.689400 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.689617 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.689676 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:29.692234 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:57569: BlockingRecv error: recv got EOF from 127.23.173.194:57569 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 14:28:29 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:29 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:29.823645 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:29.819031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:47935)
0812 14:28:29.819217 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:29.819229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:29.819303 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:29.819447 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:29.819448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:29.819627 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:29.819686 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:29.819689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.820445 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.820446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:29.820988 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:29.820989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.821323 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.821324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:29.821370 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:29.821832 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:29.821837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:29.823573 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
W20260812 14:28:29.823768 7325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:29.819120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:47935 (local address 127.23.173.254:46653)
0812 14:28:29.819184 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:29.819187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:29.819297 (+ 110us) server_negotiation.cc:407] Connection header received
0812 14:28:29.819368 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:29.819368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:29.819382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:29.819431 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:29.819850 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.820356 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.821073 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:29.821265 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:29.821320 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:29.823675 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:47935: BlockingRecv error: recv got EOF from 127.23.173.195:47935 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 12 14:28:30 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:30 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:30.576452 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:30.571872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:55407)
0812 14:28:30.572001 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:30.572012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:30.572069 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:30.572175 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:30.572176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:30.572330 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:30.572379 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:30.572380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.573125 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.573126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:30.573657 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:30.573659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.573987 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.573988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:30.574023 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:30.574474 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:30.574479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:30.576334 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260812 14:28:30.576486 7332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:30.571934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:55407 (local address 127.23.173.254:46653)
0812 14:28:30.572070 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:30.572071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:30.572080 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:30.572105 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:30.572106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:30.572116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:30.572169 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:30.572466 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.573058 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.573730 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.573934 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.573986 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:30.576428 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:55407: BlockingRecv error: recv got EOF from 127.23.173.193:55407 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
Aug 12 14:28:30 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:30 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:30.697726 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:30.692707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:37777)
0812 14:28:30.692856 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:30.692870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:30.692944 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:30.693092 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:30.693094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:30.693273 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:30.693322 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:30.693324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.693980 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.693981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:30.694576 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:30.694580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.695003 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.695005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:30.695073 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:30.695752 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:30.695758 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:30.697659 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260812 14:28:30.697841 7332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:30.692782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:37777 (local address 127.23.173.254:46653)
0812 14:28:30.692861 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:30.692863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:30.692932 (+ 69us) server_negotiation.cc:407] Connection header received
0812 14:28:30.693003 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:30.693004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:30.693018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:30.693066 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:30.693398 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.693905 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.694651 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.694934 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.694993 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:30.697769 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:37777: BlockingRecv error: recv got EOF from 127.23.173.194:37777 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 14:28:30 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:30 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:30.829834 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:30.824200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:47477)
0812 14:28:30.824402 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:30.824415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:30.824491 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:30.824633 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:30.824634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:30.824857 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:30.824924 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:30.824927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.825662 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.825663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:30.826220 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:30.826222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.826795 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.826796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:30.826865 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:30.827447 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:30.827452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:30.829767 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
W20260812 14:28:30.829964 7332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:30.824301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:47477 (local address 127.23.173.254:46653)
0812 14:28:30.824360 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:30.824363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:30.824505 (+ 142us) server_negotiation.cc:407] Connection header received
0812 14:28:30.824551 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:30.824553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:30.824568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:30.824615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:30.825006 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.825522 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.826343 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:30.826656 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:30.826777 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:30.829901 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:47477: BlockingRecv error: recv got EOF from 127.23.173.195:47477 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:28:31 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:31 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:31.581493 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:31.576922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:43663)
0812 14:28:31.577055 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:31.577065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:31.577114 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:31.577241 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:31.577242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:31.577407 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:31.577457 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:31.577458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.578171 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.578172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:31.578701 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:31.578703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.579079 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.579080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:31.579128 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:31.579600 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:31.579605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:31.581431 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260812 14:28:31.581631 7337 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:31.576982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:43663 (local address 127.23.173.254:46653)
0812 14:28:31.577119 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:31.577120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:31.577150 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:28:31.577172 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:31.577172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:31.577182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:31.577230 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:31.577557 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.578089 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.578786 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.579019 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.579074 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:31.581559 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:43663: BlockingRecv error: recv got EOF from 127.23.173.193:43663 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
Aug 12 14:28:31 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:31 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:31.702972 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:31.698223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:35619)
0812 14:28:31.698385 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:31.698399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:31.698463 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:31.698600 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:31.698601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:31.698793 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:31.698843 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:31.698845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.699522 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.699523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:31.700093 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:31.700096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.700430 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.700431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:31.700486 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:31.700948 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:31.700953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:31.702908 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
W20260812 14:28:31.703111 7337 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:31.698394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:35619 (local address 127.23.173.254:46653)
0812 14:28:31.698462 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:31.698464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:31.698472 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:28:31.698526 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:31.698526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:31.698538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:31.698589 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:31.698937 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.699446 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.700173 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.700367 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.700458 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:31.703042 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:35619: BlockingRecv error: recv got EOF from 127.23.173.194:35619 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 14:28:31 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:31 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:31.835352 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:31.830439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:47641)
0812 14:28:31.830635 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:31.830655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:31.830731 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:31.830871 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:31.830872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:31.831048 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:31.831099 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:31.831101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.831786 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.831787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:31.832338 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:31.832339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.832741 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.832742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:31.832799 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:31.833280 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:31.833285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:31.835275 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
W20260812 14:28:31.835516 7337 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:31.830597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:47641 (local address 127.23.173.254:46653)
0812 14:28:31.830680 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:31.830682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:31.830731 (+ 49us) server_negotiation.cc:407] Connection header received
0812 14:28:31.830794 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:31.830795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:31.830808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:31.830851 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:31.831158 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.831656 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.832448 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:31.832650 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:31.832709 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:31.835443 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:47641: BlockingRecv error: recv got EOF from 127.23.173.195:47641 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 14:28:32 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:32 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:32.586547 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:32.582022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:35151)
0812 14:28:32.582161 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:32.582170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:32.582214 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:32.582397 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:32.582398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:32.582571 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:32.582620 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:32.582621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.583245 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.583246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:32.583878 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:32.583881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.584214 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.584214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:32.584251 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:32.584691 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:32.584696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:32.586468 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
W20260812 14:28:32.586642 7343 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:32.582129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:35151 (local address 127.23.173.254:46653)
0812 14:28:32.582254 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:32.582256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:32.582263 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:28:32.582286 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:32.582287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:32.582297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:32.582360 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:32.582677 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.583179 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.583954 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.584156 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.584211 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:32.586579 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:35151: BlockingRecv error: recv got EOF from 127.23.173.193:35151 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
Aug 12 14:28:32 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:32 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:32.708511 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:32.703540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:60621)
0812 14:28:32.703749 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:32.703762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:32.703832 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:32.704027 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:32.704029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:32.704216 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:32.704268 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:32.704270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.704997 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.704998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:32.705583 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:32.705585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.705982 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.705983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:32.706030 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:32.706479 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:32.706484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:32.708430 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
W20260812 14:28:32.708741 7343 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:32.703646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60621 (local address 127.23.173.254:46653)
0812 14:28:32.703768 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:32.703770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:32.703815 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:28:32.703901 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:32.703902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:32.703915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:32.703969 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:32.704378 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.704888 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.705689 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.705912 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.705975 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:32.708691 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:60621: BlockingRecv error: recv got EOF from 127.23.173.194:60621 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 12 14:28:32 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:32 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:32.840602 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:32.835925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:34435)
0812 14:28:32.836068 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:32.836103 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:32.836182 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:32.836332 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:32.836334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:32.836512 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:32.836565 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:32.836568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.837238 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.837239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:32.837779 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:32.837781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.838116 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.838117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:32.838175 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:32.838646 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:32.838650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:32.840518 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260812 14:28:32.840709 7343 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:32.836058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:34435 (local address 127.23.173.254:46653)
0812 14:28:32.836144 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:32.836146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:32.836163 (+ 17us) server_negotiation.cc:407] Connection header received
0812 14:28:32.836248 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:32.836249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:32.836264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:32.836313 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:32.836641 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.837160 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.837864 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:32.838054 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:32.838112 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:32.840628 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:34435: BlockingRecv error: recv got EOF from 127.23.173.195:34435 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 14:28:33 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:33 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:33.592173 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:33.587114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:60931)
0812 14:28:33.587271 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:33.587286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:33.587353 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:33.587610 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:33.587611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:33.587862 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:33.587924 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:33.587927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.588587 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.588588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:33.589124 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:33.589126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.589560 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.589561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:33.589622 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:33.590092 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:33.590097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:33.592102 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
W20260812 14:28:33.592410 7348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:33.587231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:60931 (local address 127.23.173.254:46653)
0812 14:28:33.587343 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:33.587345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:33.587357 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:28:33.587462 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:33.587463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:33.587478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:33.587556 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:33.588003 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.588512 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.589221 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.589438 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.589507 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:33.592328 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:60931: BlockingRecv error: recv got EOF from 127.23.173.193:60931 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
Aug 12 14:28:33 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:33 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:33.713829 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:33.709024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:42743)
0812 14:28:33.709185 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:33.709197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:33.709256 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:33.709440 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:33.709441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:33.709615 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:33.709669 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:33.709670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.710386 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.710387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:33.710988 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:33.710990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.711316 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.711317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:33.711353 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:33.711854 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:33.711860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:33.713766 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
W20260812 14:28:33.713945 7348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:33.709126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:42743 (local address 127.23.173.254:46653)
0812 14:28:33.709185 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:33.709187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:33.709251 (+ 64us) server_negotiation.cc:407] Connection header received
0812 14:28:33.709314 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:33.709315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:33.709325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:33.709408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:33.709779 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.710297 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.711065 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.711261 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.711311 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:33.713881 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:42743: BlockingRecv error: recv got EOF from 127.23.173.194:42743 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 14:28:33 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:33 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:33.845927 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:33.841262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:36153)
0812 14:28:33.841486 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:33.841499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:33.841578 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:33.841709 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:33.841710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:33.841905 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:33.841957 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:33.841958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.842639 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.842640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:33.843177 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:33.843178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.843513 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.843514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:33.843570 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:33.844029 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:33.844034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:33.845853 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":138,"threads_started":1}
W20260812 14:28:33.846026 7348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:33.841354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:36153 (local address 127.23.173.254:46653)
0812 14:28:33.841480 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:33.841482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:33.841568 (+ 86us) server_negotiation.cc:407] Connection header received
0812 14:28:33.841633 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:33.841634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:33.841650 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:33.841698 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:33.842035 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.842555 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.843257 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:33.843448 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:33.843504 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:33.845966 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:36153: BlockingRecv error: recv got EOF from 127.23.173.195:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 12 14:28:34 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:34 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:34.601926 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:34.592693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:55921)
0812 14:28:34.592836 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:34.592847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:34.592897 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:34.593035 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:34.593035 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:34.593226 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:34.593277 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:34.593278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.593967 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.593968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:34.594522 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:34.594524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.594857 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.594858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:34.594895 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:34.599985 (+ 5090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:34.599991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:34.601860 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
W20260812 14:28:34.602073 7355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:34.592792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:55921 (local address 127.23.173.254:46653)
0812 14:28:34.592917 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:34.592919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:34.592927 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:28:34.592950 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:34.592951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:34.592961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:34.593024 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:34.593333 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.593859 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.594598 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.594799 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.594853 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:34.601997 (+ 7144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:55921: BlockingRecv error: recv got EOF from 127.23.173.193:55921 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
Aug 12 14:28:34 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:34 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:34.718935 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:34.714335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:41177)
0812 14:28:34.714506 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:34.714518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:34.714589 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:34.714725 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:34.714726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:34.714891 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:34.714945 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:34.714947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.715758 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.715759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:34.716308 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:34.716310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.716644 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.716645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:34.716681 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:34.717150 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:34.717155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:34.718869 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
W20260812 14:28:34.719058 7355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:34.714466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:41177 (local address 127.23.173.254:46653)
0812 14:28:34.714519 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:34.714521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:34.714573 (+ 52us) server_negotiation.cc:407] Connection header received
0812 14:28:34.714651 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:34.714652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:34.714665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:34.714702 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:34.715001 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.715659 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.716378 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.716585 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.716638 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:34.718996 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:41177: BlockingRecv error: recv got EOF from 127.23.173.194:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 14:28:34 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:34 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:34.850841 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:34.846388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:57921)
0812 14:28:34.846551 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:34.846563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:34.846629 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:34.846756 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:34.846756 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:34.846933 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:34.846986 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:34.846989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.847626 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.847628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:34.848222 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:34.848225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.848576 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.848577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:34.848617 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:34.849045 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:34.849049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:34.850762 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260812 14:28:34.850917 7355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:34.846503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:57921 (local address 127.23.173.254:46653)
0812 14:28:34.846572 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:34.846574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:34.846619 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:28:34.846678 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:34.846679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:34.846692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:34.846743 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:34.847041 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.847542 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.848316 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:34.848513 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:34.848574 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:34.850863 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:57921: BlockingRecv error: recv got EOF from 127.23.173.195:57921 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 14:28:35 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:35 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:35.606927 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:35.602425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:52063)
0812 14:28:35.602563 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:35.602574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:35.602624 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:35.602795 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:35.602796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:35.602964 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:35.603015 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:35.603017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.603657 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.603657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:35.604201 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:35.604202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.604525 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.604526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:35.604561 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:35.604987 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:35.604992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:35.606838 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
W20260812 14:28:35.607007 7364 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:35.602540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:52063 (local address 127.23.173.254:46653)
0812 14:28:35.602659 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:35.602662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:35.602670 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:28:35.602696 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:35.602697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:35.602708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:35.602779 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:35.603069 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.603597 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.604257 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.604473 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.604522 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:35.606944 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:52063: BlockingRecv error: recv got EOF from 127.23.173.193:52063 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
Aug 12 14:28:35 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:35 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:35.724227 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:35.719565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:36881)
0812 14:28:35.719769 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:35.719783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:35.719855 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:35.720009 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:35.720010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:35.720188 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:35.720242 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:35.720244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.720924 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.720925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:35.721443 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:35.721445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.721798 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.721798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:35.721836 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:35.722300 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:35.722305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:35.724167 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":125,"threads_started":1}
W20260812 14:28:35.724356 7364 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:35.719632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:36881 (local address 127.23.173.254:46653)
0812 14:28:35.719697 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:35.719716 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:35.719835 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:28:35.719925 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:35.719925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:35.719940 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:35.720000 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:35.720336 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.720840 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.721534 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.721737 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.721795 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:35.724288 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:36881: BlockingRecv error: recv got EOF from 127.23.173.194:36881 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:28:35 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:35 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:35.856048 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:35.851357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:41177)
0812 14:28:35.851513 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:35.851527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:35.851611 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:35.851789 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:35.851790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:35.851988 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:35.852042 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:35.852044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.852711 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.852712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:35.853240 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:35.853242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.853588 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.853590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:35.853646 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:35.854098 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:35.854103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:35.855965 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
W20260812 14:28:35.856137 7364 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:35.851498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:41177 (local address 127.23.173.254:46653)
0812 14:28:35.851558 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:35.851560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:35.851587 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:28:35.851673 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:35.851674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:35.851689 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:35.851770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:35.852128 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.852633 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.853325 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:35.853523 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:35.853580 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:35.856083 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:41177: BlockingRecv error: recv got EOF from 127.23.173.195:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 14:28:36 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:36 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:36.611831 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:36.607383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:53169)
0812 14:28:36.607530 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:36.607543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:36.607596 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:36.607757 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:36.607758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:36.607940 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:36.607991 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:36.607992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.608645 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.608646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:36.609198 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:36.609200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.609487 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.609488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:36.609525 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:36.609983 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:36.609988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:36.611768 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
W20260812 14:28:36.611984 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:36.607440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:53169 (local address 127.23.173.254:46653)
0812 14:28:36.607572 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:36.607574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:36.607584 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:36.607642 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:36.607642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:36.607653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:36.607724 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:36.608050 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.608550 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.609257 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.609441 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.609482 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:36.611902 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:53169: BlockingRecv error: recv got EOF from 127.23.173.193:53169 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
Aug 12 14:28:36 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:36 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:36.729600 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:36.724721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:50041)
0812 14:28:36.724865 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:36.724879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:36.724962 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:36.725100 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:36.725101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:36.725288 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:36.725338 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:36.725340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.726084 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.726085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:36.726862 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:36.726866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.727280 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.727281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:36.727327 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:36.727837 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:36.727842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:36.729537 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260812 14:28:36.729712 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:36.724829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:50041 (local address 127.23.173.254:46653)
0812 14:28:36.724887 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:36.724889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:36.724948 (+ 59us) server_negotiation.cc:407] Connection header received
0812 14:28:36.725025 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:36.725026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:36.725039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:36.725088 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:36.725408 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.725986 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.726995 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.727203 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.727269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:36.729639 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:50041: BlockingRecv error: recv got EOF from 127.23.173.194:50041 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 14:28:36 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:36 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:36.861297 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:36.856588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:46607)
0812 14:28:36.856743 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:36.856757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:36.856827 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:36.856998 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:36.856999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:36.857175 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:36.857227 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:36.857229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.857908 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.857909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:36.858444 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:36.858445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.858776 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.858778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:36.858839 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:36.859300 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:36.859304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:36.861224 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
W20260812 14:28:36.861421 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:36.856694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:46607 (local address 127.23.173.254:46653)
0812 14:28:36.856760 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:36.856762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:36.856807 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:28:36.856890 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:36.856891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:36.856907 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:36.856977 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:36.857290 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.857828 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.858509 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:36.858690 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:36.858739 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:36.861329 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:46607: BlockingRecv error: recv got EOF from 127.23.173.195:46607 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 14:28:37 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:37 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:37.617087 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:37.612281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:57459)
0812 14:28:37.612452 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:37.612465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:37.612543 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:37.612672 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:37.612672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:37.612846 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:37.612895 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:37.612896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.613574 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.613575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:37.614145 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:37.614147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.614535 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.614537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:37.614591 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:37.615152 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:37.615157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:37.616988 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
W20260812 14:28:37.617154 7379 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:37.612375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:57459 (local address 127.23.173.254:46653)
0812 14:28:37.612526 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:37.612528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:37.612538 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:37.612592 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:37.612592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:37.612602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:37.612661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:37.612962 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.613493 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.614219 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.614456 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.614512 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:37.617092 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:57459: BlockingRecv error: recv got EOF from 127.23.173.193:57459 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
Aug 12 14:28:37 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:37 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:37.734628 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:37.730035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:60783)
0812 14:28:37.730175 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:37.730187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:37.730251 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:37.730372 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:37.730372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:37.730543 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:37.730593 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:37.730595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.731198 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.731198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:37.731748 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:37.731749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.732101 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.732101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:37.732136 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:37.732569 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:37.732573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:37.734528 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260812 14:28:37.734683 7379 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:37.730097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60783 (local address 127.23.173.254:46653)
0812 14:28:37.730155 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:37.730157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:37.730241 (+ 84us) server_negotiation.cc:407] Connection header received
0812 14:28:37.730300 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:37.730301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:37.730311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:37.730357 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:37.730645 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.731144 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.731816 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.732038 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.732103 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:37.734629 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:60783: BlockingRecv error: recv got EOF from 127.23.173.194:60783 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 14:28:37 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:37 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:37.866818 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:37.861844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:44617)
0812 14:28:37.862014 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:37.862029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:37.862118 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:37.862257 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:37.862258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:37.862455 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:37.862506 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:37.862507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.863405 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.863406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:37.863990 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:37.863992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.864344 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.864346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:37.864402 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:37.864865 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:37.864869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:37.866721 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
W20260812 14:28:37.866884 7379 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:37.861953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:44617 (local address 127.23.173.254:46653)
0812 14:28:37.862041 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:37.862043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:37.862109 (+ 66us) server_negotiation.cc:407] Connection header received
0812 14:28:37.862184 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:37.862185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:37.862199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:37.862251 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:37.862591 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.863335 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.864079 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:37.864279 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:37.864338 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:37.866829 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:44617: BlockingRecv error: recv got EOF from 127.23.173.195:44617 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 14:28:38 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:38 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:38.622624 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:38.617648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:59649)
0812 14:28:38.617783 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:38.617795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:38.617884 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:38.618067 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:38.618068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:38.618246 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:38.618301 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:38.618302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.619076 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.619077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:38.619678 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:38.619681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.620154 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.620155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:38.620206 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:38.620684 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:38.620688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:38.622559 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260812 14:28:38.622761 7386 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:38.617760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:59649 (local address 127.23.173.254:46653)
0812 14:28:38.617907 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:38.617909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:38.617919 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:38.617948 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:38.617948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:38.617960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:38.618020 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:38.618418 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.618997 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.619807 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.620060 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.620129 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:38.622677 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:59649: BlockingRecv error: recv got EOF from 127.23.173.193:59649 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
Aug 12 14:28:38 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:38 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:38.739756 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:38.735104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:37877)
0812 14:28:38.735252 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:38.735265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:38.735347 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:38.735485 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:38.735487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:38.735677 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:38.735766 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:38.735769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.736434 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.736435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:38.736971 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:38.736972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.737319 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.737319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:38.737376 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:38.737847 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:38.737852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:38.739664 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260812 14:28:38.739902 7386 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:38.735191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:37877 (local address 127.23.173.254:46653)
0812 14:28:38.735256 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:38.735258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:38.735315 (+ 57us) server_negotiation.cc:407] Connection header received
0812 14:28:38.735413 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:38.735414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:38.735427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:38.735471 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:38.735850 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.736358 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.737053 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.737256 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.737313 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:38.739830 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:37877: BlockingRecv error: recv got EOF from 127.23.173.194:37877 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 14:28:38 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:38 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:38.872565 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:38.867431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:40257)
0812 14:28:38.867592 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:38.867603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:38.867669 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:38.867817 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:38.867818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:38.868000 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:38.868052 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:38.868054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.868719 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.868720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:38.869283 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:38.869286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.869763 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.869764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:38.869823 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:38.870334 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:38.870340 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:38.872499 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
W20260812 14:28:38.872730 7386 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:38.867594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:40257 (local address 127.23.173.254:46653)
0812 14:28:38.867662 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:38.867664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:38.867675 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:28:38.867738 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:38.867739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:38.867752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:38.867800 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:38.868114 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.868624 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.869456 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:38.869660 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:38.869725 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:38.872648 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:40257: BlockingRecv error: recv got EOF from 127.23.173.195:40257 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 14:28:39 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:39 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:39.631336 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:39.623114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:47535)
0812 14:28:39.623246 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:39.623257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:39.623309 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:39.623436 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:39.623437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:39.623615 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:39.623667 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:39.623668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.624468 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.624469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:39.625018 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:39.625020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.625453 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.625455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:39.625510 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:39.629300 (+ 3790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:39.629306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:39.631271 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
W20260812 14:28:39.631500 7390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:39.623161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:47535 (local address 127.23.173.254:46653)
0812 14:28:39.623292 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:39.623294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:39.623303 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:39.623364 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:39.623365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:39.623376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:39.623425 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:39.623776 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.624328 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.625129 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.625339 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.625425 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:39.631416 (+ 5991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:47535: BlockingRecv error: recv got EOF from 127.23.173.193:47535 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
Aug 12 14:28:39 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:39 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:39.744911 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:39.740261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:45363)
0812 14:28:39.740428 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:39.740438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:39.740491 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:39.740617 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:39.740617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:39.740792 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:39.740844 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:39.740845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.741520 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.741521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:39.742077 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:39.742079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.742419 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.742421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:39.742482 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:39.742941 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:39.742945 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:39.744812 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
W20260812 14:28:39.744982 7390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:39.740434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:45363 (local address 127.23.173.254:46653)
0812 14:28:39.740493 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:39.740495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:39.740506 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:28:39.740538 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:39.740539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:39.740551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:39.740607 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:39.740924 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.741428 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.742145 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.742328 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.742380 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:39.744919 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:45363: BlockingRecv error: recv got EOF from 127.23.173.194:45363 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 14:28:39 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:39 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:39.877965 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:39.873078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:59637)
0812 14:28:39.873224 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:39.873238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:39.873316 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:39.873474 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:39.873475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:39.873667 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:39.873724 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:39.873726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.874463 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.874464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:39.875013 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:39.875015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.875361 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.875362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:39.875420 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:39.875936 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:39.875940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:39.877840 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20260812 14:28:39.878011 7390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:39.873203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:59637 (local address 127.23.173.254:46653)
0812 14:28:39.873257 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:39.873259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:39.873295 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:28:39.873381 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:39.873383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:39.873398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:39.873451 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:39.873805 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.874377 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.875096 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:39.875284 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:39.875353 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:39.877955 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:59637: BlockingRecv error: recv got EOF from 127.23.173.195:59637 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 14:28:40 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:40 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:40.636379 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:40.631866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:41219)
0812 14:28:40.632093 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:40.632106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:40.632166 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:40.632294 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:40.632295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:40.632493 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:40.632541 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:40.632543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.633235 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.633236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:40.633806 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:40.633808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.634134 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.634134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:40.634171 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:40.634600 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:40.634604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:40.636307 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":115,"threads_started":1}
W20260812 14:28:40.636490 7399 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:40.631924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:41219 (local address 127.23.173.254:46653)
0812 14:28:40.632068 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:40.632070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:40.632164 (+ 94us) server_negotiation.cc:407] Connection header received
0812 14:28:40.632221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:40.632221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:40.632232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:40.632281 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:40.632640 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.633156 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.633882 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.634080 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.634132 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:40.636420 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:41219: BlockingRecv error: recv got EOF from 127.23.173.193:41219 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
Aug 12 14:28:40 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:40 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:40.749918 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:40.745318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:54209)
0812 14:28:40.745479 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:40.745491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:40.745584 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:40.745723 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:40.745725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:40.745915 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:40.745964 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:40.745966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.746633 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.746634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:40.747175 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:40.747177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.747513 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.747514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:40.747580 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:40.748048 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:40.748053 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:40.749842 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
W20260812 14:28:40.750010 7399 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:40.745408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:54209 (local address 127.23.173.254:46653)
0812 14:28:40.745479 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:40.745481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:40.745577 (+ 96us) server_negotiation.cc:407] Connection header received
0812 14:28:40.745653 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:40.745654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:40.745667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:40.745711 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:40.746053 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.746558 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.747257 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.747451 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.747508 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:40.749942 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:54209: BlockingRecv error: recv got EOF from 127.23.173.194:54209 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 14:28:40 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:40 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.195@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:40.883198 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:40.878556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.195:60937)
0812 14:28:40.878718 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:40.878732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:40.878800 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:40.878940 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:40.878941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:40.879125 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:40.879178 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:40.879180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.879906 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.879907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:40.880436 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:40.880437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.880853 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.880854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:40.880901 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:40.881366 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:40.881370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:40.883135 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
W20260812 14:28:40.883296 7399 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:40.878692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60937 (local address 127.23.173.254:46653)
0812 14:28:40.878765 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:40.878767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:40.878786 (+ 19us) server_negotiation.cc:407] Connection header received
0812 14:28:40.878849 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:40.878850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:40.878867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:40.878924 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:40.879272 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.879842 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.880505 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:40.880789 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:40.880850 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:40.883232 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.195:60937: BlockingRecv error: recv got EOF from 127.23.173.195:60937 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 14:28:41 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:41 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.193@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:41.641372 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:41.636894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.193:53321)
0812 14:28:41.637033 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:41.637044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:41.637100 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:41.637223 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:41.637224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:41.637397 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:41.637448 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:41.637450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.638142 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.638143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:41.638710 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:41.638712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.639112 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.639113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:41.639170 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:41.639614 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:41.639619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:41.641304 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260812 14:28:41.641482 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:41.636973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:53321 (local address 127.23.173.254:46653)
0812 14:28:41.637112 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:41.637114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:41.637124 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:41.637147 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:41.637148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:41.637158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:41.637213 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:41.637541 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.638064 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.638797 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.639015 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.639070 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:41.641404 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.193:53321: BlockingRecv error: recv got EOF from 127.23.173.193:53321 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
Aug 12 14:28:41 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:28:41 dist-test-slave-1l3l krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.194@KRBTEST.COM for oryx/127.23.173.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:28:41.754854 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:41.750465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46653 (local address 127.23.173.194:37977)
0812 14:28:41.750629 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:41.750639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:41.750695 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:41.750821 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:41.750822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:41.750994 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:41.751046 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:41.751048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.751748 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.751749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:41.752279 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:41.752281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.752620 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.752621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:41.752661 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:41.753116 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:41.753121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:41.754794 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.254:46653: Server oryx/127.23.173.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
W20260812 14:28:41.754945 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:41.750545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:37977 (local address 127.23.173.254:46653)
0812 14:28:41.750610 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:41.750612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:41.750682 (+ 70us) server_negotiation.cc:407] Connection header received
0812 14:28:41.750757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:41.750758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:41.750769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:41.750810 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:41.751134 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.751653 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.752348 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.752560 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.752615 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:41.754886 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.194:37977: BlockingRecv error: recv got EOF from 127.23.173.194:37977 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:28:41.798188 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 6616
I20260812 14:28:41.817582 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 6877
I20260812 14:28:41.827656 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 7085
I20260812 14:28:41.836645 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 6493
2026-08-12T14:28:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17651 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T14:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:41Z Disabled control of system clock
I20260812 14:28:41.871549 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37909
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 14:28:41.952822 7411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:41.953030 7411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:41.953049 7411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:41.954205 7411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:41.954245 7411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:41.954257 7411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:41.954279 7411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:41.954303 7411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:41.955636 7411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:41.955855 7411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:41.956115 7411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:41.961197 7424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:41.961180 7411 server_base.cc:1034] running on GCE node
W20260812 14:28:41.961261 7427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:41.961184 7430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:41.961712 7411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:41.961997 7411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:41.963155 7411 hybrid_clock.cc:648] HybridClock initialized: now 1786544921963135 us; error 31 us; skew 500 ppm
I20260812 14:28:41.964402 7411 webserver.cc:460] Webserver started at http://127.23.173.254:41717/ using document root <none> and password file <none>
I20260812 14:28:41.964596 7411 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:41.964636 7411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:41.964731 7411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:41.965766 7411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "edc6dd1146a149c1899319582387b057"
format_stamp: "Formatted at 2026-08-12 14:28:41 on dist-test-slave-1l3l"
server_key: "63b738878c58eb61d17fe6c50a3340c9"
server_key_iv: "cf13b0a3f14dcd176eaf5c2953d35dbf"
server_key_version: "encryptionkey@0"
I20260812 14:28:41.966091 7411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "edc6dd1146a149c1899319582387b057"
format_stamp: "Formatted at 2026-08-12 14:28:41 on dist-test-slave-1l3l"
server_key: "63b738878c58eb61d17fe6c50a3340c9"
server_key_iv: "cf13b0a3f14dcd176eaf5c2953d35dbf"
server_key_version: "encryptionkey@0"
I20260812 14:28:41.967351 7411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:41.968182 7442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:41.968371 7411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:41.968480 7411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "edc6dd1146a149c1899319582387b057"
format_stamp: "Formatted at 2026-08-12 14:28:41 on dist-test-slave-1l3l"
server_key: "63b738878c58eb61d17fe6c50a3340c9"
server_key_iv: "cf13b0a3f14dcd176eaf5c2953d35dbf"
server_key_version: "encryptionkey@0"
I20260812 14:28:41.968544 7411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:41.975219 7411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:41.975526 7411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:41.975658 7411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:41.979427 7411 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:37909
I20260812 14:28:41.979453 7534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:37909 every 8 connection(s)
I20260812 14:28:41.979836 7411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:28:41.980548 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:41.983326 7535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057: Bootstrap starting.
I20260812 14:28:41.983961 7535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:41.984247 7535 log.cc:824] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:41.984937 7535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057: No bootstrap required, opened a new log
I20260812 14:28:41.985771 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 7411
I20260812 14:28:41.985877 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:28:41.986063 24247 external_mini_cluster.cc:1428] Setting key 499d12ada672c14bfb55ccef20196ae3
I20260812 14:28:41.986781 7535 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc6dd1146a149c1899319582387b057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37909 } }
I20260812 14:28:41.986928 7535 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:41.986961 7535 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc6dd1146a149c1899319582387b057, State: Initialized, Role: FOLLOWER
I20260812 14:28:41.987092 7535 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc6dd1146a149c1899319582387b057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37909 } }
I20260812 14:28:41.987207 7535 raft_consensus.cc:388] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:28:41.987280 7535 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:28:41.987339 7535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:41.987946 7535 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc6dd1146a149c1899319582387b057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37909 } }
I20260812 14:28:41.988049 7535 leader_election.cc:302] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edc6dd1146a149c1899319582387b057; no voters:
I20260812 14:28:41.988211 7535 leader_election.cc:288] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:28:41.988260 7542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:41.988440 7542 raft_consensus.cc:686] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [term 1 LEADER]: Becoming Leader. State: Replica: edc6dd1146a149c1899319582387b057, State: Running, Role: LEADER
I20260812 14:28:41.988468 7535 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:28:41.989164 7542 consensus_queue.cc:237] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc6dd1146a149c1899319582387b057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37909 } }
I20260812 14:28:41.989780 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc6dd1146a149c1899319582387b057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc6dd1146a149c1899319582387b057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37909 } } }
I20260812 14:28:41.989988 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:41.990098 7543 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc6dd1146a149c1899319582387b057. Latest consensus state: current_term: 1 leader_uuid: "edc6dd1146a149c1899319582387b057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc6dd1146a149c1899319582387b057" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37909 } } }
I20260812 14:28:41.990324 7543 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:41.990317 7562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:28:41.990957 7562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:28:41.992187 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:41.986595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.23.173.254:37909)
0812 14:28:41.986813 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:41.986820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:41.986846 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:28:41.986941 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:41.986946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:41.987008 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:41.987157 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:41.987402 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.988932 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.990023 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:41.990626 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:41.990718 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:41.991432 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:41.991436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:41.991436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:41.991661 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:41.991735 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:41.991753 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:41.991819 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:41.991898 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:28:41.991976 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260812 14:28:41.996196 7562 catalog_manager.cc:1347] Generated new cluster ID: 165b91d40f404265ab35d552b7daf5e9
I20260812 14:28:41.996263 7562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:28:42.001945 7562 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:28:42.002729 7562 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:28:42.010469 7562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edc6dd1146a149c1899319582387b057: Generated new TSK 0
I20260812 14:28:42.010689 7562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:28:42.018267 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37909
--builtin_ntp_servers=127.23.173.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:28:42.128314 7571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:42.128522 7571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:42.128552 7571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:42.129710 7571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:42.129755 7571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:42.129786 7571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:28:42.131181 7571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:42.131422 7571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:42.131743 7571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:42.137619 7577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.137610 7579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.137773 7571 server_base.cc:1034] running on GCE node
W20260812 14:28:42.137981 7583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.138206 7571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:42.138438 7571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:42.139595 7571 hybrid_clock.cc:648] HybridClock initialized: now 1786544922139560 us; error 38 us; skew 500 ppm
I20260812 14:28:42.140921 7571 webserver.cc:460] Webserver started at http://127.23.173.193:35183/ using document root <none> and password file <none>
I20260812 14:28:42.141160 7571 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:42.141212 7571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:42.141347 7571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:42.142419 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "38b9ac2465f64ed5ac889e23218c2761"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "1b4a76fbfd102a74f307d839fa967738"
server_key_iv: "97d24ad8ddbc478ea43b823afff5f48e"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.142765 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "38b9ac2465f64ed5ac889e23218c2761"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "1b4a76fbfd102a74f307d839fa967738"
server_key_iv: "97d24ad8ddbc478ea43b823afff5f48e"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.144122 7571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:42.144897 7594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.145051 7571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:42.145139 7571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "38b9ac2465f64ed5ac889e23218c2761"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "1b4a76fbfd102a74f307d839fa967738"
server_key_iv: "97d24ad8ddbc478ea43b823afff5f48e"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.145210 7571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:42.157563 7571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:42.157884 7571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:42.158020 7571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:42.158237 7571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:42.158562 7571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:42.158604 7571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.158634 7571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:42.158651 7571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.164341 7571 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:45847
I20260812 14:28:42.164382 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:45847 every 8 connection(s)
I20260812 14:28:42.164744 7571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:28:42.165071 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 7571
I20260812 14:28:42.165170 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:28:42.165383 24247 external_mini_cluster.cc:1428] Setting key 31605cd1d73a005ed92df213d0bc5d12
I20260812 14:28:42.166375 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37909
--builtin_ntp_servers=127.23.173.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 14:28:42.171008 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.165489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:46893 (local address 127.23.173.254:37909)
0812 14:28:42.165559 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.165561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.165822 (+ 261us) server_negotiation.cc:407] Connection header received
0812 14:28:42.166476 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.166478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.166494 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.166559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.167247 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.167953 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.169389 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.169590 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.169662 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:42.170740 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:42.170741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:42.170741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:42.170746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:42.170761 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:42.170765 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:42.170808 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:42.170885 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 14:28:42.170927 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:28:42.171290 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.165472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37909 (local address 127.23.173.193:46893)
0812 14:28:42.165695 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:42.165741 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:42.166345 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:42.166652 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:42.166655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:42.166664 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:42.167165 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:42.167170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.168071 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.168073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:42.169277 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:42.169281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.169681 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.169683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:42.169840 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:42.170546 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:28:42.170547 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:42.170623 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:28:42.170659 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:42.170814 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:42.170818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:42.170822 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:28:42.170890 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:28:42.171011 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260812 14:28:42.171926 7790 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37909
I20260812 14:28:42.172020 7790 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:42.172205 7790 heartbeater.cc:511] Master 127.23.173.254:37909 requested a full tablet report, sending...
I20260812 14:28:42.172739 7464 ts_manager.cc:194] Registered new tserver with Master: 38b9ac2465f64ed5ac889e23218c2761 (127.23.173.193:45847)
I20260812 14:28:42.173308 7464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.193:46893
W20260812 14:28:42.251911 7794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:42.252151 7794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:42.252183 7794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:42.253369 7794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:42.253417 7794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:42.253450 7794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:28:42.254853 7794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:42.255108 7794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:42.255430 7794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:42.260599 7801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.260607 7802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.260624 7794 server_base.cc:1034] running on GCE node
W20260812 14:28:42.260607 7805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.261096 7794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:42.261344 7794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:42.262537 7794 hybrid_clock.cc:648] HybridClock initialized: now 1786544922262522 us; error 30 us; skew 500 ppm
I20260812 14:28:42.264019 7794 webserver.cc:460] Webserver started at http://127.23.173.194:46257/ using document root <none> and password file <none>
I20260812 14:28:42.264240 7794 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:42.264290 7794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:42.264423 7794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:42.265569 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "4617cc03c10b4709884f02e561f5eaeb"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "3e7cc70553294f4e367fc3e7a5716dc8"
server_key_iv: "5bfb2e03f19b6e254d415cbf65b0b580"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.265918 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "4617cc03c10b4709884f02e561f5eaeb"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "3e7cc70553294f4e367fc3e7a5716dc8"
server_key_iv: "5bfb2e03f19b6e254d415cbf65b0b580"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.267309 7794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:42.268186 7812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.268348 7794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:28:42.268435 7794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "4617cc03c10b4709884f02e561f5eaeb"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "3e7cc70553294f4e367fc3e7a5716dc8"
server_key_iv: "5bfb2e03f19b6e254d415cbf65b0b580"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.268505 7794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:42.289534 7794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:42.289887 7794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:42.290052 7794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:42.290313 7794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:42.290658 7794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:42.290694 7794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.290723 7794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:42.290740 7794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.296375 7794 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:36065
I20260812 14:28:42.296440 7990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:36065 every 8 connection(s)
I20260812 14:28:42.296777 7794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:28:42.302198 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.297639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:46267 (local address 127.23.173.254:37909)
0812 14:28:42.297721 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.297723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.297935 (+ 212us) server_negotiation.cc:407] Connection header received
0812 14:28:42.298476 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.298477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.298490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.298544 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.299264 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.299777 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.300775 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.300994 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.301050 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:42.301913 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:42.301915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:42.301915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:42.301920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:42.301937 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:42.301942 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:42.301995 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:42.302078 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:28:42.302112 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:28:42.302415 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.297564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37909 (local address 127.23.173.194:46267)
0812 14:28:42.297809 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:42.297846 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:42.298367 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:42.298631 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:42.298635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:42.298644 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:42.299164 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:42.299169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.299873 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.299874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:42.300678 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:42.300680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.301079 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.301081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:42.301211 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:42.301764 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:28:42.301764 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:42.301796 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:28:42.301820 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:42.302000 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:42.302006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:42.302009 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:28:42.302072 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:28:42.302186 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260812 14:28:42.302891 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 7794
I20260812 14:28:42.302939 7992 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37909
I20260812 14:28:42.302994 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:28:42.303059 7992 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:42.303226 24247 external_mini_cluster.cc:1428] Setting key 1456ed2f790365641c55e9cd8f5b47e2
I20260812 14:28:42.303274 7992 heartbeater.cc:511] Master 127.23.173.254:37909 requested a full tablet report, sending...
I20260812 14:28:42.303678 7463 ts_manager.cc:194] Registered new tserver with Master: 4617cc03c10b4709884f02e561f5eaeb (127.23.173.194:36065)
I20260812 14:28:42.304190 7463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.194:46267
I20260812 14:28:42.304303 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37909
--builtin_ntp_servers=127.23.173.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:28:42.389842 7995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:42.390069 7995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:42.390098 7995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:42.391309 7995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:42.391353 7995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:42.391395 7995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:28:42.392781 7995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:42.393028 7995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:42.393332 7995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:42.398626 8004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.398696 8002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.398630 8001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.398649 7995 server_base.cc:1034] running on GCE node
I20260812 14:28:42.399060 7995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:42.399326 7995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:42.400476 7995 hybrid_clock.cc:648] HybridClock initialized: now 1786544922400452 us; error 34 us; skew 500 ppm
I20260812 14:28:42.401837 7995 webserver.cc:460] Webserver started at http://127.23.173.195:37029/ using document root <none> and password file <none>
I20260812 14:28:42.402078 7995 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:42.402127 7995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:42.402247 7995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:42.403354 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "4888ce384a5d405b8b6aaf35bc047087"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "3b1d321194a172da4935b7e935b01c28"
server_key_iv: "37340dc64007ae4a8405b3b970b73606"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.403738 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "4888ce384a5d405b8b6aaf35bc047087"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "3b1d321194a172da4935b7e935b01c28"
server_key_iv: "37340dc64007ae4a8405b3b970b73606"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.405175 7995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:42.406026 8015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.406216 7995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:42.406316 7995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "4888ce384a5d405b8b6aaf35bc047087"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "3b1d321194a172da4935b7e935b01c28"
server_key_iv: "37340dc64007ae4a8405b3b970b73606"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.406378 7995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:42.420715 7995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:42.421041 7995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:42.421154 7995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:42.421353 7995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:42.421664 7995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:42.421697 7995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.421720 7995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:42.421730 7995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.427277 7995 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:39783
I20260812 14:28:42.427323 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:39783 every 8 connection(s)
I20260812 14:28:42.427651 7995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:28:42.430234 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 7995
I20260812 14:28:42.430321 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:28:42.430541 24247 external_mini_cluster.cc:1428] Setting key 1137183bbe8b58f0631f9dc31f9a3602
I20260812 14:28:42.433269 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.428561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:39983 (local address 127.23.173.254:37909)
0812 14:28:42.428623 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.428625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.428866 (+ 241us) server_negotiation.cc:407] Connection header received
0812 14:28:42.429401 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.429402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.429418 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.429468 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.430167 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.430778 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.431935 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.432139 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.432198 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:42.432999 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:42.433000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:42.433001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:42.433006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:42.433022 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:42.433026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:42.433071 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:42.433152 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 14:28:42.433185 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:28:42.433528 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.428530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37909 (local address 127.23.173.195:39983)
0812 14:28:42.428737 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:42.428774 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:42.429290 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:42.429550 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:42.429554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:42.429562 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:42.430071 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:42.430075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.430859 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.430861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:42.431846 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:42.431849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:42.432216 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:42.432217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:42.432347 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:42.432859 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:28:42.432860 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:42.432890 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:28:42.432916 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:42.433079 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:42.433086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:42.433089 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:28:42.433150 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:28:42.433262 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260812 14:28:42.434095 8205 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37909
I20260812 14:28:42.434192 8205 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:42.434373 8205 heartbeater.cc:511] Master 127.23.173.254:37909 requested a full tablet report, sending...
I20260812 14:28:42.434798 7463 ts_manager.cc:194] Registered new tserver with Master: 4888ce384a5d405b8b6aaf35bc047087 (127.23.173.195:39783)
I20260812 14:28:42.435186 7463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.195:39983
I20260812 14:28:42.442783 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:28:42.444773 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:42.444253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.23.173.254:37909)
0812 14:28:42.444299 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.444301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.444451 (+ 150us) server_negotiation.cc:407] Connection header received
0812 14:28:42.444483 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.444484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.444498 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.444556 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.444710 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41296: BlockingRecv error: recv got EOF from 127.0.0.1:41296 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:28:42.444911 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 7571
I20260812 14:28:42.480280 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 7794
I20260812 14:28:42.496809 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 7995
I20260812 14:28:42.506246 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 7411
2026-08-12T14:28:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (668 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T14:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:42Z Disabled control of system clock
I20260812 14:28:42.539093 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:32891
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:28:42.620190 8223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:42.620398 8223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:42.620417 8223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:42.621505 8223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:42.621537 8223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:42.621548 8223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:42.621560 8223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:42.621570 8223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:42.622885 8223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:32891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:42.623104 8223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:42.623354 8223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:42.628154 8233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.628207 8230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.628206 8229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.628244 8223 server_base.cc:1034] running on GCE node
I20260812 14:28:42.628612 8223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:42.628898 8223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:42.630045 8223 hybrid_clock.cc:648] HybridClock initialized: now 1786544922630029 us; error 30 us; skew 500 ppm
I20260812 14:28:42.631245 8223 webserver.cc:460] Webserver started at http://127.23.173.254:39321/ using document root <none> and password file <none>
I20260812 14:28:42.631469 8223 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:42.631518 8223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:42.631649 8223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:42.632750 8223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "7bcc02e1a26f46debb42f0c6d060aad4"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "0ee5d047924c80e5a6095d598e641ccc"
server_key_iv: "648ed28a5ccb6d6d6f5d2e6b199e1ee2"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.633096 8223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "7bcc02e1a26f46debb42f0c6d060aad4"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "0ee5d047924c80e5a6095d598e641ccc"
server_key_iv: "648ed28a5ccb6d6d6f5d2e6b199e1ee2"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.634320 8223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:28:42.634966 8242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.635128 8223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:42.635210 8223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "7bcc02e1a26f46debb42f0c6d060aad4"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "0ee5d047924c80e5a6095d598e641ccc"
server_key_iv: "648ed28a5ccb6d6d6f5d2e6b199e1ee2"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.635269 8223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:42.638531 8223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:42.638772 8223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:42.638891 8223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:42.642573 8223 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:32891
I20260812 14:28:42.642606 8328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:32891 every 8 connection(s)
I20260812 14:28:42.642905 8223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:28:42.643293 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 8223
I20260812 14:28:42.643390 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:28:42.643615 24247 external_mini_cluster.cc:1428] Setting key 24cffa6db866aacf8c237773a44e36e6
I20260812 14:28:42.643631 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:42.646310 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.644337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.23.173.254:32891)
0812 14:28:42.644538 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.644547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.644579 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:28:42.645045 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.645055 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.645108 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.645271 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.645874 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:42.645877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:42.645877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:42.645882 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:42.645927 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:42.645941 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:42.646013 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:42.646066 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 14:28:42.646121 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260812 14:28:42.646778 8330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4: Bootstrap starting.
I20260812 14:28:42.647410 8330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:42.647647 8330 log.cc:824] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:42.648294 8330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4: No bootstrap required, opened a new log
I20260812 14:28:42.649538 8330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 32891 } }
I20260812 14:28:42.649662 8330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:42.649690 8330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcc02e1a26f46debb42f0c6d060aad4, State: Initialized, Role: FOLLOWER
I20260812 14:28:42.649780 8330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 32891 } }
I20260812 14:28:42.649847 8330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:28:42.649873 8330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:28:42.649904 8330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:42.650382 8330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 32891 } }
I20260812 14:28:42.650468 8330 leader_election.cc:302] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bcc02e1a26f46debb42f0c6d060aad4; no voters:
I20260812 14:28:42.650589 8330 leader_election.cc:288] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:28:42.650830 8337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:42.650868 8330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:28:42.651001 8337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [term 1 LEADER]: Becoming Leader. State: Replica: 7bcc02e1a26f46debb42f0c6d060aad4, State: Running, Role: LEADER
I20260812 14:28:42.651137 8337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 32891 } }
I20260812 14:28:42.651489 8341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 32891 } } }
I20260812 14:28:42.651523 8342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bcc02e1a26f46debb42f0c6d060aad4. Latest consensus state: current_term: 1 leader_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcc02e1a26f46debb42f0c6d060aad4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 32891 } } }
I20260812 14:28:42.651588 8341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:42.651595 8342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:42.651816 8349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:28:42.652256 8349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:28:42.653788 8349 catalog_manager.cc:1347] Generated new cluster ID: 5fa0f9dbf1a34f77ac08ff94dec4e476
I20260812 14:28:42.653829 8349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:28:42.659616 8349 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:28:42.659780 8349 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:28:42.668040 8349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bcc02e1a26f46debb42f0c6d060aad4: Generated new TSK 0
I20260812 14:28:42.668200 8349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:28:42.673220 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:32891
--builtin_ntp_servers=127.23.173.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:28:42.785771 8368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:42.785976 8368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:42.785997 8368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:42.787102 8368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:42.787139 8368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:42.787163 8368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:28:42.788491 8368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:42.788717 8368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:42.789000 8368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:42.793898 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.794006 8368 server_base.cc:1034] running on GCE node
W20260812 14:28:42.793866 8378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.793848 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.794364 8368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:42.794591 8368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:42.795744 8368 hybrid_clock.cc:648] HybridClock initialized: now 1786544922795728 us; error 32 us; skew 500 ppm
I20260812 14:28:42.796963 8368 webserver.cc:460] Webserver started at http://127.23.173.193:41367/ using document root <none> and password file <none>
I20260812 14:28:42.797173 8368 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:42.797221 8368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:42.797349 8368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:42.798386 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "9a29a1f22c3e4a7b806a07e75f8388dc"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "2280377d9170a019005b838df580bb3e"
server_key_iv: "5ba1b251ae90498c05c56e4da6a11ade"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.798722 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "9a29a1f22c3e4a7b806a07e75f8388dc"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "2280377d9170a019005b838df580bb3e"
server_key_iv: "5ba1b251ae90498c05c56e4da6a11ade"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.799966 8368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:28:42.800688 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.800846 8368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:42.800931 8368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "9a29a1f22c3e4a7b806a07e75f8388dc"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "2280377d9170a019005b838df580bb3e"
server_key_iv: "5ba1b251ae90498c05c56e4da6a11ade"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.801000 8368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:42.804216 8368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:42.804457 8368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:42.804576 8368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:42.804773 8368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:42.805104 8368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:42.805142 8368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.805168 8368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:42.805192 8368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.810957 8368 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:33599
I20260812 14:28:42.810993 8577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:33599 every 8 connection(s)
I20260812 14:28:42.811282 8368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:28:42.813916 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.811954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:49361 (local address 127.23.173.254:32891)
0812 14:28:42.811995 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.811997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.812217 (+ 220us) server_negotiation.cc:407] Connection header received
0812 14:28:42.812642 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.812643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.812649 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.812705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.813755 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:42.813756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:42.813756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:42.813759 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:42.813770 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:42.813773 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:42.813810 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:42.813855 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 14:28:42.813865 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:28:42.814102 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.811936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:32891 (local address 127.23.173.193:49361)
0812 14:28:42.812108 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:42.812145 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:42.812564 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:42.812752 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:42.812755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:42.812763 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:42.813631 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:28:42.813632 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:42.813659 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:28:42.813688 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:42.813810 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:42.813812 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:42.813814 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:28:42.813861 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:28:42.813912 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260812 14:28:42.814600 8579 heartbeater.cc:347] Connected to a master server at 127.23.173.254:32891
I20260812 14:28:42.814690 8579 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:42.814831 8579 heartbeater.cc:511] Master 127.23.173.254:32891 requested a full tablet report, sending...
I20260812 14:28:42.815294 8270 ts_manager.cc:194] Registered new tserver with Master: 9a29a1f22c3e4a7b806a07e75f8388dc (127.23.173.193:33599)
I20260812 14:28:42.819334 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 8368
I20260812 14:28:42.819422 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:28:42.819590 24247 external_mini_cluster.cc:1428] Setting key 08aa1d57bb5a8a332a71a9a7dfaa9114
I20260812 14:28:42.820415 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:32891
--builtin_ntp_servers=127.23.173.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:28:42.900805 8585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:42.900995 8585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:42.901015 8585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:42.902112 8585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:42.902150 8585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:42.902175 8585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:28:42.903538 8585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:42.903764 8585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:42.904031 8585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:42.908810 8594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.908810 8592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:42.908861 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:42.909111 8585 server_base.cc:1034] running on GCE node
I20260812 14:28:42.909291 8585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:42.909505 8585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:42.910635 8585 hybrid_clock.cc:648] HybridClock initialized: now 1786544922910620 us; error 34 us; skew 500 ppm
I20260812 14:28:42.911821 8585 webserver.cc:460] Webserver started at http://127.23.173.194:39091/ using document root <none> and password file <none>
I20260812 14:28:42.912032 8585 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:42.912076 8585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:42.912201 8585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:42.913194 8585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "91584f97d829470582f70ac7f6472d29"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "fa6c2b0769d0ff0f972a9bc324c4428d"
server_key_iv: "b7534bafde0088d5b566438559cc048e"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.913523 8585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "91584f97d829470582f70ac7f6472d29"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "fa6c2b0769d0ff0f972a9bc324c4428d"
server_key_iv: "b7534bafde0088d5b566438559cc048e"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.914710 8585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:42.915326 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.915498 8585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:42.915576 8585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "91584f97d829470582f70ac7f6472d29"
format_stamp: "Formatted at 2026-08-12 14:28:42 on dist-test-slave-1l3l"
server_key: "fa6c2b0769d0ff0f972a9bc324c4428d"
server_key_iv: "b7534bafde0088d5b566438559cc048e"
server_key_version: "encryptionkey@0"
I20260812 14:28:42.915644 8585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:42.918797 8585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:42.919028 8585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:42.919154 8585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:42.919340 8585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:42.919620 8585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:42.919652 8585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.919677 8585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:42.919693 8585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:42.925083 8585 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:33035
I20260812 14:28:42.925148 8803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:33035 every 8 connection(s)
I20260812 14:28:42.925403 8585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:28:42.928042 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.926027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:51533 (local address 127.23.173.254:32891)
0812 14:28:42.926075 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:42.926077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:42.926297 (+ 220us) server_negotiation.cc:407] Connection header received
0812 14:28:42.926736 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:42.926737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:42.926744 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:42.926779 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:42.927872 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:42.927873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:42.927873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:42.927877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:42.927892 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:42.927896 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:42.927933 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:42.927977 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:28:42.927988 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:28:42.928205 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:42.926006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:32891 (local address 127.23.173.194:51533)
0812 14:28:42.926194 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:42.926225 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:42.926644 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:42.926835 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:42.926839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:42.926846 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:42.927699 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:28:42.927699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:42.927763 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:28:42.927795 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:42.927932 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:42.927934 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:42.927936 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:28:42.927983 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:28:42.928026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260812 14:28:42.928642 8804 heartbeater.cc:347] Connected to a master server at 127.23.173.254:32891
I20260812 14:28:42.928718 8804 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:42.928853 8804 heartbeater.cc:511] Master 127.23.173.254:32891 requested a full tablet report, sending...
I20260812 14:28:42.929196 8270 ts_manager.cc:194] Registered new tserver with Master: 91584f97d829470582f70ac7f6472d29 (127.23.173.194:33035)
I20260812 14:28:42.935334 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 8585
I20260812 14:28:42.935441 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:28:42.935627 24247 external_mini_cluster.cc:1428] Setting key d046012d43fad525bd00b1e90eee68a7
I20260812 14:28:42.936527 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:32891
--builtin_ntp_servers=127.23.173.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:28:43.017081 8809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:43.017285 8809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:43.017314 8809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:43.018501 8809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:43.018543 8809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:43.018576 8809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:28:43.019951 8809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:43.020190 8809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:43.020462 8809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:43.025202 8815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:43.025288 8816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:43.025430 8821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:43.025938 8809 server_base.cc:1034] running on GCE node
I20260812 14:28:43.026158 8809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:43.026367 8809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:43.029577 8809 hybrid_clock.cc:648] HybridClock initialized: now 1786544923029545 us; error 36 us; skew 500 ppm
I20260812 14:28:43.031114 8809 webserver.cc:460] Webserver started at http://127.23.173.195:38055/ using document root <none> and password file <none>
I20260812 14:28:43.031368 8809 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:43.031419 8809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:43.031551 8809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:43.032856 8809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "2b0b868e365746698dd2b42d087c9168"
format_stamp: "Formatted at 2026-08-12 14:28:43 on dist-test-slave-1l3l"
server_key: "1c18c4688748702dc4b233dbebed6a0f"
server_key_iv: "6b7cfebd6c441de3c93b350a983602c3"
server_key_version: "encryptionkey@0"
I20260812 14:28:43.033258 8809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "2b0b868e365746698dd2b42d087c9168"
format_stamp: "Formatted at 2026-08-12 14:28:43 on dist-test-slave-1l3l"
server_key: "1c18c4688748702dc4b233dbebed6a0f"
server_key_iv: "6b7cfebd6c441de3c93b350a983602c3"
server_key_version: "encryptionkey@0"
I20260812 14:28:43.034523 8809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:43.035207 8827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:43.035380 8809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:43.035465 8809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "2b0b868e365746698dd2b42d087c9168"
format_stamp: "Formatted at 2026-08-12 14:28:43 on dist-test-slave-1l3l"
server_key: "1c18c4688748702dc4b233dbebed6a0f"
server_key_iv: "6b7cfebd6c441de3c93b350a983602c3"
server_key_version: "encryptionkey@0"
I20260812 14:28:43.035534 8809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:43.038934 8809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:43.039178 8809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:43.039300 8809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:43.039491 8809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:43.039826 8809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:43.039863 8809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:43.039891 8809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:43.039908 8809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:43.045471 8809 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:37525
I20260812 14:28:43.045527 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:37525 every 8 connection(s)
I20260812 14:28:43.045826 8809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:28:43.048537 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:43.046449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60455 (local address 127.23.173.254:32891)
0812 14:28:43.046491 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:43.046493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:43.046701 (+ 208us) server_negotiation.cc:407] Connection header received
0812 14:28:43.047143 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:43.047144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:43.047150 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:43.047202 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:43.048360 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:43.048361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:43.048361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:28:43.048365 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:43.048380 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:28:43.048384 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:43.048429 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:43.048473 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:28:43.048484 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:28:43.048712 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:43.046425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:32891 (local address 127.23.173.195:60455)
0812 14:28:43.046596 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:43.046630 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:43.047055 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:43.047251 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:43.047254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:43.047261 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:43.048226 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:28:43.048227 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:43.048255 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:28:43.048284 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:43.048428 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:43.048430 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:43.048432 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:28:43.048479 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:28:43.048522 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260812 14:28:43.049165 9014 heartbeater.cc:347] Connected to a master server at 127.23.173.254:32891
I20260812 14:28:43.049247 9014 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:43.049357 9014 heartbeater.cc:511] Master 127.23.173.254:32891 requested a full tablet report, sending...
I20260812 14:28:43.049681 8270 ts_manager.cc:194] Registered new tserver with Master: 2b0b868e365746698dd2b42d087c9168 (127.23.173.195:37525)
I20260812 14:28:43.051628 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 8809
I20260812 14:28:43.051725 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:28:43.051924 24247 external_mini_cluster.cc:1428] Setting key 3632ee42ad625a07ee9819f1c1c74025
I20260812 14:28:43.052747 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:28:43.054383 8334 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:28:43.053999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.23.173.254:32891)
0812 14:28:43.054039 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:43.054040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:43.054147 (+ 107us) server_negotiation.cc:407] Connection header received
0812 14:28:43.054189 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:43.054190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:43.054193 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:43.054239 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:43.054339 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51000: BlockingRecv error: recv got EOF from 127.0.0.1:51000 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:28:43.054629 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 8368
I20260812 14:28:43.072561 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 8585
I20260812 14:28:43.078924 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 8809
I20260812 14:28:43.088124 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 8223
2026-08-12T14:28:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (583 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:28:43 dist-test-slave-1l3l krb5kdc[9031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:28:43 dist-test-slave-1l3l krb5kdc[9031](info): set up 2 sockets
Aug 12 14:28:43 dist-test-slave-1l3l krb5kdc[9031](info): commencing operation
krb5kdc: starting...
W20260812 14:28:44.269649 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:28:44.386199 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:42075
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:44.469193 9057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:44.469436 9057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:44.469471 9057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:44.470680 9057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:28:44.470727 9057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:44.470746 9057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:44.470762 9057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:28:44.470777 9057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:28:44.472419 9057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:42075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:44.472687 9057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:44.472918 9057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:44.477980 9064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:44.477979 9066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:44.478123 9063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:44.478369 9057 server_base.cc:1034] running on GCE node
I20260812 14:28:44.478646 9057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:44.478929 9057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:44.480087 9057 hybrid_clock.cc:648] HybridClock initialized: now 1786544924480069 us; error 31 us; skew 500 ppm
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:44.482985 9057 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:28:44.483513 9057 webserver.cc:460] Webserver started at http://127.23.173.254:37513/ using document root <none> and password file <none>
I20260812 14:28:44.483803 9057 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:44.483856 9057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:44.483975 9057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:44.485086 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "17fe73d030a44426a9620997fa2ddab7"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "47b29287293640e9d70469e90b028d66"
server_key_iv: "c08b1d16e90a886c4cebb55d1af60413"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.485420 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "17fe73d030a44426a9620997fa2ddab7"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "47b29287293640e9d70469e90b028d66"
server_key_iv: "c08b1d16e90a886c4cebb55d1af60413"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.486822 9057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:44.487640 9074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.487823 9057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:28:44.487914 9057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "17fe73d030a44426a9620997fa2ddab7"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "47b29287293640e9d70469e90b028d66"
server_key_iv: "c08b1d16e90a886c4cebb55d1af60413"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.487985 9057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:44.503579 9057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:44.505708 9057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:44.505860 9057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:44.510216 9057 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:42075
I20260812 14:28:44.510265 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:42075 every 8 connection(s)
I20260812 14:28:44.510629 9057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:28:44.510972 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 9057
I20260812 14:28:44.511075 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:28:44.511247 24247 external_mini_cluster.cc:1428] Setting key 6d98b8ad031c6ac3fd2e43c32128a74c
I20260812 14:28:44.511986 9160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:44.514858 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: Bootstrap starting.
I20260812 14:28:44.515581 9160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:44.515933 9160 log.cc:824] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:44.516707 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: No bootstrap required, opened a new log
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:44.518536 9160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fe73d030a44426a9620997fa2ddab7" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42075 } }
I20260812 14:28:44.518729 9160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:44.518770 9160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fe73d030a44426a9620997fa2ddab7, State: Initialized, Role: FOLLOWER
I20260812 14:28:44.518908 9160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fe73d030a44426a9620997fa2ddab7" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42075 } }
I20260812 14:28:44.518976 9160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:28:44.518998 9160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:28:44.519047 9160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:44.519680 9160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fe73d030a44426a9620997fa2ddab7" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42075 } }
I20260812 14:28:44.519816 9160 leader_election.cc:302] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17fe73d030a44426a9620997fa2ddab7; no voters:
I20260812 14:28:44.519995 9160 leader_election.cc:288] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:28:44.520052 9168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:44.520255 9160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:28:44.520382 9168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [term 1 LEADER]: Becoming Leader. State: Replica: 17fe73d030a44426a9620997fa2ddab7, State: Running, Role: LEADER
I20260812 14:28:44.520493 9168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fe73d030a44426a9620997fa2ddab7" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42075 } }
I20260812 14:28:44.521488 9171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17fe73d030a44426a9620997fa2ddab7. Latest consensus state: current_term: 1 leader_uuid: "17fe73d030a44426a9620997fa2ddab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fe73d030a44426a9620997fa2ddab7" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42075 } } }
I20260812 14:28:44.521595 9171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [sys.catalog]: This master's current role is: LEADER
I20260812 14:28:44.521953 9170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17fe73d030a44426a9620997fa2ddab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fe73d030a44426a9620997fa2ddab7" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 42075 } } }
W20260812 14:28:44.522153 9200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 14:28:44.522397 9170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7 [sys.catalog]: This master's current role is: LEADER
W20260812 14:28:44.522411 9200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:28:44.522219 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:44.511783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.23.173.254:42075)
0812 14:28:44.511972 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:44.511980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:44.512006 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:28:44.512475 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:44.512482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:44.512530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:44.512645 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:44.513434 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.514361 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.515377 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.515863 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.515986 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:44.518220 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:44.518227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:44.518227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:44.518238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:44.520028 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:44.520507 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:44.520510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:44.520511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:44.520621 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:44.520870 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:44.520871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:44.520871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:44.521240 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:44.521403 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:44.521857 (+ 454us) server_negotiation.cc:299] Negotiation successful
0812 14:28:44.522009 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260812 14:28:44.522683 9202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:28:44.522858 9202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:28:44.524349 9202 catalog_manager.cc:1347] Generated new cluster ID: a21facbcc79b4ee1beaed10efc889d9f
I20260812 14:28:44.524403 9202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:28:44.536047 9202 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:28:44.536561 9202 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:28:44.541805 9202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: Generated new TSK 0
I20260812 14:28:44.541980 9202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:28:44.576460 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42075
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:44.673103 9210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:44.673310 9210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:44.673332 9210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:44.674520 9210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:44.674561 9210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:44.674587 9210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:28:44.676123 9210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:44.676357 9210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:44.676568 9210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:44.681385 9217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:44.681393 9216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:44.681393 9219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:44.681658 9210 server_base.cc:1034] running on GCE node
I20260812 14:28:44.681859 9210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:44.682096 9210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:44.683269 9210 hybrid_clock.cc:648] HybridClock initialized: now 1786544924683247 us; error 32 us; skew 500 ppm
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:44.691732 9210 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:28:44.692198 9210 webserver.cc:460] Webserver started at http://127.23.173.193:33611/ using document root <none> and password file <none>
I20260812 14:28:44.692410 9210 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:44.692451 9210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:44.692545 9210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:44.693555 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "8b7180d2e5a641e1b74997668a27ac05"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "c77d6cc61e842e24f0e919bbf234aecb"
server_key_iv: "af5e21a2998871d16d145885329c2401"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.693867 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "8b7180d2e5a641e1b74997668a27ac05"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "c77d6cc61e842e24f0e919bbf234aecb"
server_key_iv: "af5e21a2998871d16d145885329c2401"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.695152 9210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:28:44.695926 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.696085 9210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:44.696170 9210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "8b7180d2e5a641e1b74997668a27ac05"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "c77d6cc61e842e24f0e919bbf234aecb"
server_key_iv: "af5e21a2998871d16d145885329c2401"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.696228 9210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:44.706494 9210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:44.708209 9210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:44.708318 9210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:44.708492 9210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:44.708835 9210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:44.708880 9210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.708900 9210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:44.708913 9210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.714468 9210 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:39139
I20260812 14:28:44.714509 9423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:39139 every 8 connection(s)
I20260812 14:28:44.714802 9210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:44.722860 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 9210
I20260812 14:28:44.722970 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:28:44.723192 24247 external_mini_cluster.cc:1428] Setting key ed5746ec34ae040edac33391d81e84e1
I20260812 14:28:44.723937 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:44.715518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:44789 (local address 127.23.173.254:42075)
0812 14:28:44.715601 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:44.715604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:44.715800 (+ 196us) server_negotiation.cc:407] Connection header received
0812 14:28:44.716327 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:44.716328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:44.716341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:44.716394 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:44.717331 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.717834 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.718814 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.719019 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.719076 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:44.721278 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:44.721280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:44.721280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:44.721288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:44.722469 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:44.722871 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:44.722872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:44.722873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:44.722941 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:44.723181 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:44.723182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:44.723182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:44.723326 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:44.723394 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:44.723748 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 14:28:44.723847 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:28:44.724126 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:44.715470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42075 (local address 127.23.173.193:44789)
0812 14:28:44.715653 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:44.715689 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:44.716223 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:44.716465 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:44.716468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:44.717032 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:44.717229 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:44.717235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.717922 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.717923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:44.718720 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:44.718722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.719100 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.719101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:44.719215 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:44.719679 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:44.719686 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:44.721173 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:44.722563 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:44.722571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:44.722573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:44.722798 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:44.723003 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:44.723004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:44.723004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:44.723119 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:44.723424 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:44.723431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:44.723592 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 14:28:44.723721 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 14:28:44.723895 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260812 14:28:44.724568 9424 heartbeater.cc:347] Connected to a master server at 127.23.173.254:42075
I20260812 14:28:44.724689 9424 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:44.724882 9424 heartbeater.cc:511] Master 127.23.173.254:42075 requested a full tablet report, sending...
I20260812 14:28:44.725371 9103 ts_manager.cc:194] Registered new tserver with Master: 8b7180d2e5a641e1b74997668a27ac05 (127.23.173.193:39139)
I20260812 14:28:44.725978 9103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:44789
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:28:44.750345 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42075
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:44.835937 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:44.836162 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:44.836189 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:44.837388 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:44.837435 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:44.837460 9430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:28:44.838958 9430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:44.839232 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:44.839485 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:44.844607 9440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:44.844624 9443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:44.844624 9436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:44.844815 9430 server_base.cc:1034] running on GCE node
I20260812 14:28:44.845024 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:44.845275 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:44.846467 9430 hybrid_clock.cc:648] HybridClock initialized: now 1786544924846449 us; error 32 us; skew 500 ppm
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:44.849064 9430 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:28:44.849499 9430 webserver.cc:460] Webserver started at http://127.23.173.194:38871/ using document root <none> and password file <none>
I20260812 14:28:44.849736 9430 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:44.849782 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:44.849881 9430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:44.851020 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "979d68892c86436a88de5d585304c757"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "923484a7d2d194beb61e8f2cf3c7d2ea"
server_key_iv: "68e5e9a06d97f0c7725417ef85cad35a"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.851334 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "979d68892c86436a88de5d585304c757"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "923484a7d2d194beb61e8f2cf3c7d2ea"
server_key_iv: "68e5e9a06d97f0c7725417ef85cad35a"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.852690 9430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:28:44.853545 9450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.853735 9430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:28:44.853814 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "979d68892c86436a88de5d585304c757"
format_stamp: "Formatted at 2026-08-12 14:28:44 on dist-test-slave-1l3l"
server_key: "923484a7d2d194beb61e8f2cf3c7d2ea"
server_key_iv: "68e5e9a06d97f0c7725417ef85cad35a"
server_key_version: "encryptionkey@0"
I20260812 14:28:44.853891 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:44.873926 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:44.876266 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:44.876408 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:44.876659 9430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:44.876991 9430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:44.877027 9430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.877058 9430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:44.877089 9430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:44.882728 9430 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:38717
I20260812 14:28:44.882803 9642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:38717 every 8 connection(s)
I20260812 14:28:44.883139 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:28:44.886442 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 9430
I20260812 14:28:44.886545 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:28:44.886761 24247 external_mini_cluster.cc:1428] Setting key b81eae8df8fbbe949c34a506d9edf8c0
Aug 12 14:28:44 dist-test-slave-1l3l krb5kdc[9031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544924, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:44.893371 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:44.884081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:58209 (local address 127.23.173.254:42075)
0812 14:28:44.884148 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:44.884150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:44.884356 (+ 206us) server_negotiation.cc:407] Connection header received
0812 14:28:44.884911 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:44.884912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:44.884925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:44.884987 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:44.885961 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.886585 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.887854 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.888118 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.888182 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:44.890573 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:44.890574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:44.890574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:44.890582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:44.891925 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:44.892436 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:44.892437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:44.892437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:44.892496 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:44.892761 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:44.892762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:44.892762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:44.892861 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:44.892907 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:44.893241 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 14:28:44.893306 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:28:44.893661 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:44.884045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42075 (local address 127.23.173.194:58209)
0812 14:28:44.884223 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:44.884273 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:44.884808 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:44.885060 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:44.885065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:44.885651 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:44.885857 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:44.885861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.886689 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.886691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:44.887759 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:44.887763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:44.888195 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:44.888196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:44.888311 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:44.888818 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:44.888827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:44.890480 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:44.892022 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:44.892030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:44.892033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:44.892354 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:44.892554 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:44.892555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:44.892555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:44.892700 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:44.892918 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:44.892924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:44.893106 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 14:28:44.893234 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 14:28:44.893418 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260812 14:28:44.894202 9643 heartbeater.cc:347] Connected to a master server at 127.23.173.254:42075
I20260812 14:28:44.894295 9643 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:44.894495 9643 heartbeater.cc:511] Master 127.23.173.254:42075 requested a full tablet report, sending...
I20260812 14:28:44.894878 9103 ts_manager.cc:194] Registered new tserver with Master: 979d68892c86436a88de5d585304c757 (127.23.173.194:38717)
I20260812 14:28:44.895242 9103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:58209
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:28:44.913905 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42075
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:28:44.996541 9651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:28:44.996754 9651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:28:44.996785 9651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:28:44.997920 9651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:28:44.997960 9651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:28:44.997994 9651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:28:44.999452 9651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:42075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:28:44.999698 9651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:28:44.999975 9651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:28:45.004874 9664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:28:45.004913 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:45.005075 9651 server_base.cc:1034] running on GCE node
W20260812 14:28:45.004908 9663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:28:45.005343 9651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:28:45.005581 9651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:28:45.006722 9651 hybrid_clock.cc:648] HybridClock initialized: now 1786544925006709 us; error 32 us; skew 500 ppm
Aug 12 14:28:45 dist-test-slave-1l3l krb5kdc[9031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:28:45.009212 9651 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:28:45.009615 9651 webserver.cc:460] Webserver started at http://127.23.173.195:42165/ using document root <none> and password file <none>
I20260812 14:28:45.009812 9651 fs_manager.cc:359] Metadata directory not provided
I20260812 14:28:45.009858 9651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:28:45.009982 9651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:28:45.010974 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "4f5dea8e5b2743ce898abc3a3dce452e"
format_stamp: "Formatted at 2026-08-12 14:28:45 on dist-test-slave-1l3l"
server_key: "203607256a18ff630d2b859dfc679395"
server_key_iv: "897944f8a4b095fb92bb2e5b63aa865c"
server_key_version: "encryptionkey@0"
I20260812 14:28:45.011318 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "4f5dea8e5b2743ce898abc3a3dce452e"
format_stamp: "Formatted at 2026-08-12 14:28:45 on dist-test-slave-1l3l"
server_key: "203607256a18ff630d2b859dfc679395"
server_key_iv: "897944f8a4b095fb92bb2e5b63aa865c"
server_key_version: "encryptionkey@0"
I20260812 14:28:45.012624 9651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:28:45.013290 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:45.013437 9651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:28:45.013520 9651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "4f5dea8e5b2743ce898abc3a3dce452e"
format_stamp: "Formatted at 2026-08-12 14:28:45 on dist-test-slave-1l3l"
server_key: "203607256a18ff630d2b859dfc679395"
server_key_iv: "897944f8a4b095fb92bb2e5b63aa865c"
server_key_version: "encryptionkey@0"
I20260812 14:28:45.013585 9651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:28:45.047613 9651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:28:45.049388 9651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:28:45.049526 9651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:28:45.049754 9651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:28:45.050055 9651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:28:45.050091 9651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:45.050122 9651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:28:45.050146 9651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:28:45.055892 9651 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:36397
I20260812 14:28:45.055949 9870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:36397 every 8 connection(s)
I20260812 14:28:45.056255 9651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:28:45.059478 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 9651
I20260812 14:28:45.059576 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:28:45.059793 24247 external_mini_cluster.cc:1428] Setting key 0a1c2d0f4032d5492701afb7d64db9bf
Aug 12 14:28:45 dist-test-slave-1l3l krb5kdc[9031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:28:45.065032 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.056946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:44195 (local address 127.23.173.254:42075)
0812 14:28:45.057018 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.057021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.057229 (+ 208us) server_negotiation.cc:407] Connection header received
0812 14:28:45.057707 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.057708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.057719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.057765 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:45.058719 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.059202 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.060566 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.060777 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.060825 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.062844 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:45.062846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:45.062846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:45.062852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:45.064008 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:45.064348 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:45.064349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:45.064349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:45.064394 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:45.064569 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:45.064570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:45.064570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:45.064665 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:45.064711 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.064914 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.064969 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:28:45.065220 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.056922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:42075 (local address 127.23.173.195:44195)
0812 14:28:45.057124 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.057162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:45.057630 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:45.057847 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:45.057851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:45.058449 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:28:45.058641 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.058647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.059269 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.059270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.060479 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.060482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.060850 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.060851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.060965 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.061452 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:28:45.061461 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:28:45.062761 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:28:45.064073 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:45.064086 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:45.064088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:45.064292 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:45.064436 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:28:45.064436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:28:45.064437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:28:45.064525 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:28:45.064708 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:28:45.064711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:28:45.064819 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 14:28:45.064898 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:28:45.065013 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260812 14:28:45.065626 9871 heartbeater.cc:347] Connected to a master server at 127.23.173.254:42075
I20260812 14:28:45.065713 9871 heartbeater.cc:464] Registering TS with master...
I20260812 14:28:45.065884 9871 heartbeater.cc:511] Master 127.23.173.254:42075 requested a full tablet report, sending...
I20260812 14:28:45.066222 9103 ts_manager.cc:194] Registered new tserver with Master: 4f5dea8e5b2743ce898abc3a3dce452e (127.23.173.195:36397)
I20260812 14:28:45.066579 9103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:44195
I20260812 14:28:45.071350 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:28:45.077755 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.072607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.23.173.254:42075)
0812 14:28:45.072654 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.072656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.072763 (+ 107us) server_negotiation.cc:407] Connection header received
0812 14:28:45.072808 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.072808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.072816 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.072861 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:28:45.073339 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.073777 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.074411 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.074611 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.074665 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.075723 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:28:45.075725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:28:45.075725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:28:45.075731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:28:45.076739 (+ 1008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:45.077037 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:45.077038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:45.077039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:45.077089 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:28:45.077249 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:28:45.077250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:28:45.077250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:28:45.077386 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:28:45.077475 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.077596 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.077671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:28:45.079200 9103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:28:45.080776 9103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:28:45.087217 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.083332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.23.173.193:39139)
0812 14:28:45.083558 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.083562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.083572 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:28:45.083619 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.083621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.083636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.083733 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:45.083950 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.085005 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.086066 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.086783 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.086871 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.086935 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.087087 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.087136 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":131,"threads_started":1}
I20260812 14:28:45.087216 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.082901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:38717 (local address 127.0.0.1:49230)
0812 14:28:45.083086 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.083128 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:45.083193 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:45.083864 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:45.083864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:45.083867 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:45.083916 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.083917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.084851 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.084852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.086106 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.086110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.086947 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.086948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.086997 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.086999 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:28:45.087066 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:28:45.087120 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":51,"thread_start_us":75,"threads_started":1}
I20260812 14:28:45.087216 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.082929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39139 (local address 127.0.0.1:57582)
0812 14:28:45.083183 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.083192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:45.083233 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:45.083824 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:45.083825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:45.083828 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:45.083873 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.083875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.085105 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.085106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.085994 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.085996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.086883 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.086884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.086965 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.086981 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 14:28:45.087056 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 14:28:45.087119 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260812 14:28:45.088068 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.082929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:36397 (local address 127.0.0.1:52846)
0812 14:28:45.083290 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.083298 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:45.083331 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:45.083541 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:45.083544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:45.083546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:45.083679 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.083682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.085059 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.085060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.086114 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.086115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.087820 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.087821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.087880 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.087885 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:28:45.087945 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:28:45.087983 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":13,"thread_start_us":84,"threads_started":1}
I20260812 14:28:45.088078 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.083155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.23.173.195:36397)
0812 14:28:45.083315 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.083320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.083332 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:28:45.083391 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.083393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.083409 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.083508 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:45.083791 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.084954 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.087018 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.087743 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.087818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.087872 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.087959 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.088003 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260812 14:28:45.088268 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.082997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.23.173.194:38717)
0812 14:28:45.083610 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.083614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.083632 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:28:45.083670 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.083673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.083688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.083992 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:45.084051 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.084763 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.086193 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.086872 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.086938 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.086983 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.087687 (+ 704us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.087756 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":101,"threads_started":1}
I20260812 14:28:45.088304 9314 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73ae2db978441884faed90597a8706 (DEFAULT_TABLE table=test-table [id=a6b49cda846d4d2fa395e81215118f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:45.088683 9314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73ae2db978441884faed90597a8706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:45.088832 9537 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73ae2db978441884faed90597a8706 (DEFAULT_TABLE table=test-table [id=a6b49cda846d4d2fa395e81215118f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:45.089143 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73ae2db978441884faed90597a8706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:45.091328 9899 tablet_bootstrap.cc:492] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: Bootstrap starting.
I20260812 14:28:45.091926 9900 tablet_bootstrap.cc:492] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Bootstrap starting.
I20260812 14:28:45.091887 9759 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73ae2db978441884faed90597a8706 (DEFAULT_TABLE table=test-table [id=a6b49cda846d4d2fa395e81215118f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:28:45.092164 9899 tablet_bootstrap.cc:654] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:45.092221 9759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73ae2db978441884faed90597a8706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:28:45.092598 9900 tablet_bootstrap.cc:654] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:45.092626 9899 log.cc:824] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:45.093163 9900 log.cc:824] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:45.093263 9899 tablet_bootstrap.cc:492] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: No bootstrap required, opened a new log
I20260812 14:28:45.093308 9899 ts_tablet_manager.cc:1397] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:28:45.093927 9900 tablet_bootstrap.cc:492] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: No bootstrap required, opened a new log
I20260812 14:28:45.093984 9900 ts_tablet_manager.cc:1397] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:28:45.094700 9899 raft_consensus.cc:348] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.094820 9899 raft_consensus.cc:374] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:45.094841 9899 raft_consensus.cc:729] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7180d2e5a641e1b74997668a27ac05, State: Initialized, Role: FOLLOWER
I20260812 14:28:45.094929 9899 consensus_queue.cc:260] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.095183 9899 ts_tablet_manager.cc:1428] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:28:45.095201 9424 heartbeater.cc:503] Master 127.23.173.254:42075 was elected leader, sending a full tablet report...
I20260812 14:28:45.095670 9906 tablet_bootstrap.cc:492] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Bootstrap starting.
I20260812 14:28:45.096079 9900 raft_consensus.cc:348] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.096231 9900 raft_consensus.cc:374] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:45.096292 9900 raft_consensus.cc:729] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979d68892c86436a88de5d585304c757, State: Initialized, Role: FOLLOWER
I20260812 14:28:45.096431 9906 tablet_bootstrap.cc:654] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Neither blocks nor log segments found. Creating new log.
I20260812 14:28:45.096408 9900 consensus_queue.cc:260] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.096666 9900 ts_tablet_manager.cc:1428] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:28:45.096738 9643 heartbeater.cc:503] Master 127.23.173.254:42075 was elected leader, sending a full tablet report...
I20260812 14:28:45.096978 9906 log.cc:824] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:28:45.097674 9906 tablet_bootstrap.cc:492] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: No bootstrap required, opened a new log
I20260812 14:28:45.097728 9906 ts_tablet_manager.cc:1397] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:28:45.099098 9906 raft_consensus.cc:348] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.099216 9906 raft_consensus.cc:374] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:28:45.099236 9906 raft_consensus.cc:729] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dea8e5b2743ce898abc3a3dce452e, State: Initialized, Role: FOLLOWER
I20260812 14:28:45.099321 9906 consensus_queue.cc:260] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.099606 9871 heartbeater.cc:503] Master 127.23.173.254:42075 was elected leader, sending a full tablet report...
I20260812 14:28:45.099529 9906 ts_tablet_manager.cc:1428] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260812 14:28:45.133769 9644 tablet.cc:2369] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:28:45.215577 9425 tablet.cc:2369] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:28:45.272091 9910 raft_consensus.cc:482] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:28:45.272243 9910 raft_consensus.cc:504] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.272645 9910 leader_election.cc:288] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 979d68892c86436a88de5d585304c757 (127.23.173.194:38717), 8b7180d2e5a641e1b74997668a27ac05 (127.23.173.193:39139)
I20260812 14:28:45.275569 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.272919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:40985 (local address 127.23.173.193:39139)
0812 14:28:45.272981 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.272984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.273100 (+ 116us) server_negotiation.cc:407] Connection header received
0812 14:28:45.273139 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.273141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.273154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.273212 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:45.273384 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.273864 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.274891 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.275233 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.275295 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.275306 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.275461 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.275500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:28:45.275594 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.272851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39139 (local address 127.23.173.195:40985)
0812 14:28:45.273020 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.273034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:45.273095 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:45.273226 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:45.273228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:45.273230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:45.273313 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.273316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.273943 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.273944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.274790 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.274792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.275308 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.275309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.275399 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.275402 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:28:45.275447 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:28:45.275488 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":36}
I20260812 14:28:45.275622 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.272946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60659 (local address 127.23.173.194:38717)
0812 14:28:45.273014 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.273016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.273176 (+ 160us) server_negotiation.cc:407] Connection header received
0812 14:28:45.273231 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.273232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.273244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.273300 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:28:45.273442 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.273899 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.274911 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.275293 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.275370 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.275379 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.275515 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.275556 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:28:45.275868 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.272886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:38717 (local address 127.23.173.195:60659)
0812 14:28:45.273098 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.273109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:28:45.273175 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:28:45.273305 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:28:45.273306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:28:45.273308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:28:45.273372 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.273374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.273976 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.273977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.274792 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:28:45.274795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.275379 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.275380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:28:45.275455 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.275458 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:28:45.275504 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:28:45.275542 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":2,"thread_start_us":132,"threads_started":1}
I20260812 14:28:45.275957 9348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73ae2db978441884faed90597a8706" candidate_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7180d2e5a641e1b74997668a27ac05" is_pre_election: true
I20260812 14:28:45.276131 9348 raft_consensus.cc:2454] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5dea8e5b2743ce898abc3a3dce452e in term 0.
I20260812 14:28:45.276198 9564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73ae2db978441884faed90597a8706" candidate_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979d68892c86436a88de5d585304c757" is_pre_election: true
I20260812 14:28:45.276355 9564 raft_consensus.cc:2454] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5dea8e5b2743ce898abc3a3dce452e in term 0.
I20260812 14:28:45.276341 9686 leader_election.cc:302] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5dea8e5b2743ce898abc3a3dce452e, 8b7180d2e5a641e1b74997668a27ac05; no voters:
I20260812 14:28:45.276476 9910 raft_consensus.cc:2781] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:28:45.276539 9910 raft_consensus.cc:482] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:28:45.276566 9910 raft_consensus.cc:3037] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:45.277309 9910 raft_consensus.cc:504] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.277439 9910 leader_election.cc:288] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [CANDIDATE]: Term 1 election: Requested vote from peers 979d68892c86436a88de5d585304c757 (127.23.173.194:38717), 8b7180d2e5a641e1b74997668a27ac05 (127.23.173.193:39139)
I20260812 14:28:45.277617 9348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73ae2db978441884faed90597a8706" candidate_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7180d2e5a641e1b74997668a27ac05"
I20260812 14:28:45.277688 9348 raft_consensus.cc:3037] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:45.278476 9348 raft_consensus.cc:2454] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5dea8e5b2743ce898abc3a3dce452e in term 1.
I20260812 14:28:45.278589 9564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73ae2db978441884faed90597a8706" candidate_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979d68892c86436a88de5d585304c757"
I20260812 14:28:45.278669 9686 leader_election.cc:302] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5dea8e5b2743ce898abc3a3dce452e, 8b7180d2e5a641e1b74997668a27ac05; no voters:
I20260812 14:28:45.278676 9564 raft_consensus.cc:3037] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:28:45.278762 9910 raft_consensus.cc:2781] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:28:45.278937 9910 raft_consensus.cc:686] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 1 LEADER]: Becoming Leader. State: Replica: 4f5dea8e5b2743ce898abc3a3dce452e, State: Running, Role: LEADER
I20260812 14:28:45.279132 9910 consensus_queue.cc:237] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } }
I20260812 14:28:45.279618 9564 raft_consensus.cc:2454] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5dea8e5b2743ce898abc3a3dce452e in term 1.
I20260812 14:28:45.280095 9102 catalog_manager.cc:5495] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5dea8e5b2743ce898abc3a3dce452e (127.23.173.195). New cstate: current_term: 1 leader_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5dea8e5b2743ce898abc3a3dce452e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36397 } health_report { overall_health: HEALTHY } } }
W20260812 14:28:45.306811 9873 tablet.cc:2369] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:28:45.312922 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.310646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.23.173.195:36397)
0812 14:28:45.310711 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:28:45.310713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:28:45.310722 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:28:45.310756 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:28:45.310757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:28:45.310776 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:28:45.310835 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:28:45.310974 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.311446 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.312200 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:28:45.312387 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:28:45.312442 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:28:45.312608 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:28:45.312712 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:28:45.312784 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:28:45.312822 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:28:45.312844 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:28:45.314707 9348 raft_consensus.cc:1261] T 3a73ae2db978441884faed90597a8706 P 8b7180d2e5a641e1b74997668a27ac05 [term 1 FOLLOWER]: Refusing update from remote peer 4f5dea8e5b2743ce898abc3a3dce452e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:45.314707 9564 raft_consensus.cc:1261] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 1 FOLLOWER]: Refusing update from remote peer 4f5dea8e5b2743ce898abc3a3dce452e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:28:45.315083 9913 consensus_queue.cc:1035] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7180d2e5a641e1b74997668a27ac05" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:45.315223 9910 consensus_queue.cc:1035] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [LEADER]: Connected to new peer: Peer: permanent_uuid: "979d68892c86436a88de5d585304c757" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:28:45.318224 9918 mvcc.cc:205] Tried to move back new op lower bound from 7317688014086479872 to 7317688013944049664. Current Snapshot: MvccSnapshot[applied={T|T < 7317688014086479872}]
I20260812 14:28:45.318260 9919 mvcc.cc:205] Tried to move back new op lower bound from 7317688014086479872 to 7317688013944049664. Current Snapshot: MvccSnapshot[applied={T|T < 7317688014086479872 or (T in {7317688014086479872})}]
I20260812 14:28:45.320374 9103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:45.320478 9103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:28:45.321252 9103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17fe73d030a44426a9620997fa2ddab7: Sending DeleteTablet for 3 replicas of tablet 3a73ae2db978441884faed90597a8706
I20260812 14:28:45.321602 9759 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a73ae2db978441884faed90597a8706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:45 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:52846
I20260812 14:28:45.321625 9314 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a73ae2db978441884faed90597a8706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:45 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:57582
I20260812 14:28:45.321635 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a73ae2db978441884faed90597a8706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:28:45 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:49230
I20260812 14:28:45.321918 9934 tablet_replica.cc:330] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: stopping tablet replica
I20260812 14:28:45.322050 9934 raft_consensus.cc:2229] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:28:45.322167 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 9210
I20260812 14:28:45.322181 9934 raft_consensus.cc:2258] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:45.322726 9938 tablet_replica.cc:330] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: stopping tablet replica
I20260812 14:28:45.322829 9938 raft_consensus.cc:2229] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:28:45.322925 9938 raft_consensus.cc:2258] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:28:45.327828 9938 ts_tablet_manager.cc:1905] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:45.329511 9938 ts_tablet_manager.cc:1918] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:45.329643 9938 log.cc:1196] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/3a73ae2db978441884faed90597a8706
I20260812 14:28:45.329941 9938 ts_tablet_manager.cc:1939] T 3a73ae2db978441884faed90597a8706 P 979d68892c86436a88de5d585304c757: Deleting consensus metadata
I20260812 14:28:45.330394 9079 catalog_manager.cc:4841] TS 979d68892c86436a88de5d585304c757 (127.23.173.194:38717): tablet 3a73ae2db978441884faed90597a8706 (table test-table [id=a6b49cda846d4d2fa395e81215118f58]) successfully deleted
I20260812 14:28:45.335117 9934 ts_tablet_manager.cc:1905] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:28:45.336601 9934 ts_tablet_manager.cc:1918] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:28:45.336664 9934 log.cc:1196] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/3a73ae2db978441884faed90597a8706
I20260812 14:28:45.336884 9934 ts_tablet_manager.cc:1939] T 3a73ae2db978441884faed90597a8706 P 4f5dea8e5b2743ce898abc3a3dce452e: Deleting consensus metadata
I20260812 14:28:45.337270 9080 catalog_manager.cc:4841] TS 4f5dea8e5b2743ce898abc3a3dce452e (127.23.173.195:36397): tablet 3a73ae2db978441884faed90597a8706 (table test-table [id=a6b49cda846d4d2fa395e81215118f58]) successfully deleted
W20260812 14:28:45.354926 9078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:39139 (error 108)
I20260812 14:28:45.355540 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:28:45.355398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39139 (local address 127.0.0.1:57598)
0812 14:28:45.355455 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:28:45.355500 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 14:28:45.355556 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 9430
W20260812 14:28:45.360352 9078 catalog_manager.cc:4584] TS 8b7180d2e5a641e1b74997668a27ac05 (127.23.173.193:39139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a73ae2db978441884faed90597a8706: Network error: Client connection negotiation failed: client connection to 127.23.173.193:39139: connect: Connection refused (error 111)
I20260812 14:28:45.363730 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 9651
I20260812 14:28:45.374430 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 9057
2026-08-12T14:28:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2287 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:28:45 dist-test-slave-1l3l krb5kdc[9945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:28:45 dist-test-slave-1l3l krb5kdc[9945](info): set up 2 sockets
Aug 12 14:28:45 dist-test-slave-1l3l krb5kdc[9945](info): commencing operation
krb5kdc: starting...
W20260812 14:28:46.550575 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:28:46 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:28:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/rangeradmin_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/rangeradmin_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/rangerlookup_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/rangerlookup_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
I20260812 14:28:46.714632 24247 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOhwVqn/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/postgres -l logfile start
2026-08-12 14:28:49.273 UTC [9994] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 14:28:49.273 UTC [9994] LOG: listening on IPv4 address "127.23.173.212", port 49477
2026-08-12 14:28:49.276 UTC [9994] LOG: listening on Unix socket "/tmp/.s.PGSQL.49477"
2026-08-12 14:28:49.296 UTC [9998] LOG: database system was shut down at 2026-08-12 14:28:47 UTC
2026-08-12 14:28:49.300 UTC [9994] LOG: database system is ready to accept connections
I20260812 14:28:50.390249 24247 mini_postgres.cc:96] Postgres bound to 49477
2026-08-12 14:28:50.398 UTC [10011] FATAL: database "slave" does not exist
127.23.173.212:49477 - accepting connections
I20260812 14:28:50.399377 24247 mini_ranger.cc:161] Starting Ranger...
I20260812 14:28:50.416173 24247 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 14:28:50.901558 24247 mini_ranger.cc:87] Created ranger Postgres database
I20260812 14:28:50.901638 24247 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-admin
2026-08-12 14:28:51,356 [I] DB FLAVOR :POSTGRES
2026-08-12 14:28:51,356 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:28:51,356 [I] Checking connection..
2026-08-12 14:28:51,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 14:28:51,645 [I] Checking connection passed.
2026-08-12 14:28:51,645 [I] --------- Verifying version history table ---------
2026-08-12 14:28:51,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:28:51,909 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:28:51,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:28:52,178 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:28:52,178 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 14:28:52,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 14:28:52.424 UTC [10114] WARNING: there is no transaction in progress
2026-08-12 14:28:52,434 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 14:28:52,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:28:52,703 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 14:28:52,704 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:28:52,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 14:28:52,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 14:28:53,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1l3l.c.gcp-upstream.internal','N') ;"
2026-08-12 14:28:53,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:28:53,752 [I] Table x_portal_user does not exist in database ranger
2026-08-12 14:28:53,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:28:54,017 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 14:28:54,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:28:54,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:28:54,502 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 14:28:54,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 14:28:55.479 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.497 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.513 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.530 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.547 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.569 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.637 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.644 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.659 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.669 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.680 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.693 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.703 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.709 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.716 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:55.722 UTC [10298] WARNING: there is no transaction in progress
2026-08-12 14:28:56,382 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 14:28:56,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 14:28:56,615 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 14:28:56,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:28:56,883 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 14:28:56,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:28:57,139 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 14:28:57,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:28:57,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:28:57,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1l3l.c.gcp-upstream.internal';"
2026-08-12 14:28:57,923 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 14:28:57,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 14:28:58,182 [I] DB_PATCHES have already been applied
I20260812 14:28:58.188055 24247 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOhwVqn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 14:28:58.188122 24247 mini_ranger.cc:193] Using host: 127.23.173.212
I20260812 14:28:58.191699 24247 mini_ranger.cc:227] Ranger admin URL: http://127.23.173.212:34059
Aug 12, 2026 2:28:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:28:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:28:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:28:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:28:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:28:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:28:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 14:29:00.744496 24247 mini_ranger.cc:160] Time spent starting Ranger: real 10.345s user 0.003s sys 0.010s
Aug 12, 2026 2:29:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:29:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:29:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:29:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:29:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 14:29:22.586823 24247 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/rangerkms_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/rangerkms_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.173.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 14:29:22 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544962, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 14:29:22.683586 24247 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 14:29:22.693459 24247 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 14:29:23.538009 24247 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 14:29:23.538080 24247 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms
I20260812 14:29:23.538092 24247 mini_ranger_kms.cc:227] Using postgres at 127.23.173.212:49477
2026-08-12 14:29:23,999 [I] DB FLAVOR :POSTGRES
2026-08-12 14:29:24,000 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:29:24,000 [I] Checking connection
2026-08-12 14:29:24,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 14:29:24,247 [I] connection success
2026-08-12 14:29:24,247 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 14:29:24,247 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 14:29:24,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 14:29:24,492 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 14:29:24,493 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:29:24,493 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 14:29:24,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:49477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 14:29:24,782 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 14:29:25.290191 24247 mini_ranger_kms.cc:310] Created kms service
I20260812 14:29:25.476399 24247 mini_ranger_kms.cc:326] Created kudu user
I20260812 14:29:25.519584 24247 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 14:29:25 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544962, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:29:25.935523 24247 mini_ranger_kms.cc:384] Added ranger policy
I20260812 14:29:25.935644 24247 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 14:29:25.935662 24247 mini_ranger_kms.cc:240] Using host: 127.23.173.212
I20260812 14:29:25.940843 24247 mini_ranger_kms.cc:276] Ranger KMS PID: 10597
I20260812 14:29:25.940989 24247 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.173.212:49563
Aug 12, 2026 2:29:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:29:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 14:29:28 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:29:28.493434 24247 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.810s user 0.022s sys 0.242s
I20260812 14:29:28.493566 24247 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 14:29:28.493642 24247 mini_ranger_kms.cc:401] 127.23.173.212:49563/kms/v1/keys
Aug 12 14:29:28 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.212@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
Aug 12 14:29:29 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.212@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
Aug 12 14:29:31 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:29:31.364475 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46211
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46211
--ranger_config_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:49563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:29:31.446460 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:29:31.446669 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:29:31.446691 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:29:31.447873 10675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:29:31.447917 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:29:31.447930 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:29:31.447942 10675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:29:31.447957 10675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:29:31.449631 10675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.173.212:49563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46211
--ranger_config_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:29:31.449862 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:29:31.450075 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:29:31.455013 10675 server_base.cc:1034] running on GCE node
W20260812 14:29:31.455165 10690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:31.455046 10696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:31.455011 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:29:31.455596 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:29:31.455915 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:29:31.457069 10675 hybrid_clock.cc:648] HybridClock initialized: now 1786544971457052 us; error 32 us; skew 500 ppm
Aug 12 14:29:31 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544971, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:29:31.459802 10675 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:29:31.460268 10675 webserver.cc:460] Webserver started at http://127.23.173.254:38753/ using document root <none> and password file <none>
I20260812 14:29:31.460487 10675 fs_manager.cc:359] Metadata directory not provided
I20260812 14:29:31.460526 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:29:31.460654 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:29:31 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544971, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:29:31.511909 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "164e6946a1544db2abb2f0e10019493c"
format_stamp: "Formatted at 2026-08-12 14:29:31 on dist-test-slave-1l3l"
server_key: "f0c073f73c2b9313c5fcfe487a360a90"
server_key_iv: "01db0cc4afa513da049d30a4ed764dd0"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:31.512382 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "164e6946a1544db2abb2f0e10019493c"
format_stamp: "Formatted at 2026-08-12 14:29:31 on dist-test-slave-1l3l"
server_key: "f0c073f73c2b9313c5fcfe487a360a90"
server_key_iv: "01db0cc4afa513da049d30a4ed764dd0"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:31.514128 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 14:29:31.529273 10706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:31.529492 10675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:29:31.529605 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "164e6946a1544db2abb2f0e10019493c"
format_stamp: "Formatted at 2026-08-12 14:29:31 on dist-test-slave-1l3l"
server_key: "f0c073f73c2b9313c5fcfe487a360a90"
server_key_iv: "01db0cc4afa513da049d30a4ed764dd0"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:31.529704 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:29:31.549312 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:29:31.551254 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:29:31.551404 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:29:31.556394 10675 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46211
I20260812 14:29:31.556425 10790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46211 every 8 connection(s)
I20260812 14:29:31.556816 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:29:31.557498 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:29:31.559819 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c: Bootstrap starting.
I20260812 14:29:31.560360 10793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c: Neither blocks nor log segments found. Creating new log.
I20260812 14:29:31.560629 10793 log.cc:824] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:29:31.561192 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 10675
I20260812 14:29:31.561281 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:29:31.561453 24247 external_mini_cluster.cc:1428] Setting key daea59dd1601b939efd6d462501c20ba
I20260812 14:29:31.561599 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c: No bootstrap required, opened a new log
I20260812 14:29:31.564584 10793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e6946a1544db2abb2f0e10019493c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46211 } }
I20260812 14:29:31.564790 10793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:29:31.564846 10793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164e6946a1544db2abb2f0e10019493c, State: Initialized, Role: FOLLOWER
I20260812 14:29:31.564998 10793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e6946a1544db2abb2f0e10019493c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46211 } }
I20260812 14:29:31.565100 10793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:29:31.565145 10793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:29:31.565197 10793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:29:31.565866 10793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e6946a1544db2abb2f0e10019493c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46211 } }
I20260812 14:29:31.565994 10793 leader_election.cc:302] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164e6946a1544db2abb2f0e10019493c; no voters:
I20260812 14:29:31.566180 10793 leader_election.cc:288] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:29:31.566231 10801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:29:31.566466 10801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [term 1 LEADER]: Becoming Leader. State: Replica: 164e6946a1544db2abb2f0e10019493c, State: Running, Role: LEADER
I20260812 14:29:31.566529 10793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:29:31.566609 10801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e6946a1544db2abb2f0e10019493c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46211 } }
I20260812 14:29:31.567014 10801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164e6946a1544db2abb2f0e10019493c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e6946a1544db2abb2f0e10019493c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46211 } } }
I20260812 14:29:31.567162 10801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [sys.catalog]: This master's current role is: LEADER
I20260812 14:29:31.567349 10801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164e6946a1544db2abb2f0e10019493c. Latest consensus state: current_term: 1 leader_uuid: "164e6946a1544db2abb2f0e10019493c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e6946a1544db2abb2f0e10019493c" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46211 } } }
I20260812 14:29:31.567461 10801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c [sys.catalog]: This master's current role is: LEADER
I20260812 14:29:31.567724 10817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:29:31.568292 10817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:29:31 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:29:31.570417 10817 catalog_manager.cc:1347] Generated new cluster ID: 3e98a479811d405cbb9101fa83819ff0
I20260812 14:29:31.570515 10817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:29:31.572237 10793 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:29:31.572932 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:31.561914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.23.173.254:46211)
0812 14:29:31.562080 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:31.562088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:31.562115 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:29:31.562972 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:31.562980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:31.563029 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:31.563154 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:29:31.563799 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:31.564655 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:31.565724 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:31.566321 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:31.566438 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:31.569317 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:29:31.569321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:29:31.569321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:29:31.569332 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:29:31.571167 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:31.571515 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:31.571517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:31.571518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:31.571618 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:31.571803 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:31.571804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:31.571804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:31.572064 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:29:31.572200 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:31.572632 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 14:29:31.572735 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260812 14:29:31.584858 10817 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:29:31.585637 10817 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:29:31.598166 10817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c: Generated new TSK 0
I20260812 14:29:31.598413 10817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:29:32.324882 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46211
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:29:32.446548 10874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:29:32.446841 10874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:29:32.446877 10874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:29:32.448670 10874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:29:32.448740 10874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:29:32.448781 10874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:29:32.450989 10874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.173.212:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:29:32.451285 10874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:29:32.451609 10874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:29:32.457261 10885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:32.457261 10883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:29:32.457463 10874 server_base.cc:1034] running on GCE node
W20260812 14:29:32.457674 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:29:32.457911 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:29:32.458248 10874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:29:32.460431 10874 hybrid_clock.cc:648] HybridClock initialized: now 1786544972460393 us; error 53 us; skew 500 ppm
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:29:32.463883 10874 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:29:32.464519 10874 webserver.cc:460] Webserver started at http://127.23.173.193:41449/ using document root <none> and password file <none>
I20260812 14:29:32.464825 10874 fs_manager.cc:359] Metadata directory not provided
I20260812 14:29:32.464898 10874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:29:32.465042 10874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:29:32.480959 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "b09feac6583247ccbd0f6d935c4ffb15"
format_stamp: "Formatted at 2026-08-12 14:29:32 on dist-test-slave-1l3l"
server_key: "c9c4849f6e87c40072a2613f34634fec"
server_key_iv: "0079ff19979431d44089543593da5236"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:32.481419 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "b09feac6583247ccbd0f6d935c4ffb15"
format_stamp: "Formatted at 2026-08-12 14:29:32 on dist-test-slave-1l3l"
server_key: "c9c4849f6e87c40072a2613f34634fec"
server_key_iv: "0079ff19979431d44089543593da5236"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:32.484983 10874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 14:29:32.501019 10902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:32.501202 10874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:29:32.501293 10874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "b09feac6583247ccbd0f6d935c4ffb15"
format_stamp: "Formatted at 2026-08-12 14:29:32 on dist-test-slave-1l3l"
server_key: "c9c4849f6e87c40072a2613f34634fec"
server_key_iv: "0079ff19979431d44089543593da5236"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:32.501400 10874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:29:32.518986 10874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:29:32.521600 10874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:29:32.521836 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:29:32.522119 10874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:29:32.522548 10874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:29:32.522629 10874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:32.522687 10874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:29:32.522730 10874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:32.530035 10874 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:40565
I20260812 14:29:32.530599 10874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:29:32.532501 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:40565 every 8 connection(s)
I20260812 14:29:32.535034 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 10874
I20260812 14:29:32.535137 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:29:32.535334 24247 external_mini_cluster.cc:1428] Setting key e3eeaeb544adee2a58884b151e4965c6
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
I20260812 14:29:32.566367 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:32.533579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:54961 (local address 127.23.173.254:46211)
0812 14:29:32.539534 (+ 5955us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:32.539537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:32.539549 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:29:32.539577 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:32.539578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:32.539592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:32.540595 (+ 1003us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:29:32.540644 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.541435 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.543964 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.544289 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.547111 (+ 2822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:32.547194 (+ 83us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:29:32.547195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:29:32.547195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:29:32.547201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:29:32.549155 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:32.549698 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:32.549700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:32.549700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:32.549772 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:32.550112 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:32.550113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:32.550114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:32.552679 (+ 2565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:29:32.552762 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:32.566105 (+ 13343us) server_negotiation.cc:299] Negotiation successful
0812 14:29:32.566248 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5906,"thread_start_us":96,"threads_started":1}
I20260812 14:29:32.567325 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:32.533558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46211 (local address 127.23.173.193:54961)
0812 14:29:32.536100 (+ 2542us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:32.536155 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:32.538878 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:32.539834 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:32.539840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:32.540139 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:29:32.540466 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:32.540474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.541582 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.541584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:32.542953 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:32.542957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.544411 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.544413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:32.544588 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:32.545251 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:29:32.545262 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:29:32.547015 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:29:32.549257 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:29:32.549265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:29:32.549269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:29:32.549586 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:29:32.549856 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:29:32.549857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:29:32.549858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:29:32.550012 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:29:32.564721 (+ 14709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:29:32.564737 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:29:32.564979 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 14:29:32.566652 (+ 1673us) client_negotiation.cc:240] Negotiation successful
0812 14:29:32.566869 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2485,"thread_start_us":95,"threads_started":1}
I20260812 14:29:32.568297 11068 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46211
I20260812 14:29:32.568398 11068 heartbeater.cc:464] Registering TS with master...
I20260812 14:29:32.568599 11068 heartbeater.cc:511] Master 127.23.173.254:46211 requested a full tablet report, sending...
I20260812 14:29:32.569212 10736 ts_manager.cc:194] Registered new tserver with Master: b09feac6583247ccbd0f6d935c4ffb15 (127.23.173.193:40565)
I20260812 14:29:32.570156 10736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:54961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:29:32.576658 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46211
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 14:29:32.702982 11077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:29:32.703279 11077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:29:32.703315 11077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:29:32.705034 11077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:29:32.705094 11077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:29:32.705132 11077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:29:32.707387 11077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.173.212:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:29:32.707827 11077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:29:32.708104 11077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:29:32.714406 11087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:32.714426 11086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:32.714787 11091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:29:32.715276 11077 server_base.cc:1034] running on GCE node
I20260812 14:29:32.715497 11077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:29:32.715792 11077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:29:32.716950 11077 hybrid_clock.cc:648] HybridClock initialized: now 1786544972716932 us; error 34 us; skew 500 ppm
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:29:32.720242 11077 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:29:32.720791 11077 webserver.cc:460] Webserver started at http://127.23.173.194:42069/ using document root <none> and password file <none>
I20260812 14:29:32.721045 11077 fs_manager.cc:359] Metadata directory not provided
I20260812 14:29:32.721096 11077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:29:32.721220 11077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:29:32.737640 11077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "a11c06b03eaa4861a52a5a53fa6f4d58"
format_stamp: "Formatted at 2026-08-12 14:29:32 on dist-test-slave-1l3l"
server_key: "0eeb00b683b4458644585ce0be47fa4e"
server_key_iv: "41b9f7761f3480450c082c0f3467aeeb"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:32.738114 11077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "a11c06b03eaa4861a52a5a53fa6f4d58"
format_stamp: "Formatted at 2026-08-12 14:29:32 on dist-test-slave-1l3l"
server_key: "0eeb00b683b4458644585ce0be47fa4e"
server_key_iv: "41b9f7761f3480450c082c0f3467aeeb"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:32.741086 11077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 14:29:32.764120 11102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:32.764398 11077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 14:29:32.764497 11077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "a11c06b03eaa4861a52a5a53fa6f4d58"
format_stamp: "Formatted at 2026-08-12 14:29:32 on dist-test-slave-1l3l"
server_key: "0eeb00b683b4458644585ce0be47fa4e"
server_key_iv: "41b9f7761f3480450c082c0f3467aeeb"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:32.764602 11077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:29:32.777503 11077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:29:32.779894 11077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:29:32.780032 11077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:29:32.780247 11077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:29:32.780586 11077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:29:32.780638 11077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:32.780666 11077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:29:32.780722 11077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:32.786762 11077 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:44057
I20260812 14:29:32.787232 11077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:29:32.788622 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 11077
I20260812 14:29:32.788748 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:29:32.788976 24247 external_mini_cluster.cc:1428] Setting key 24c12a9ca99e6fac6e7276ca946dd064
I20260812 14:29:32.797801 11288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:44057 every 8 connection(s)
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
I20260812 14:29:32.811409 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:32.798765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:59973 (local address 127.23.173.254:46211)
0812 14:29:32.798818 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:32.798821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:32.799101 (+ 280us) server_negotiation.cc:407] Connection header received
0812 14:29:32.799836 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:32.799838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:32.799857 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:32.799890 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:29:32.800751 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.801506 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.803028 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.803339 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.803383 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:32.807824 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:29:32.807826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:29:32.807827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:29:32.807835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:29:32.809700 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:32.810227 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:32.810228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:32.810229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:32.810292 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:32.810598 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:32.810599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:32.810600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:32.810746 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:29:32.810794 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:32.811212 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 14:29:32.811310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:29:32.811968 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:32.798463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46211 (local address 127.23.173.194:59973)
0812 14:29:32.798968 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:32.799016 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:32.799692 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:32.800088 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:32.800094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:32.800340 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:29:32.800632 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:32.800639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.801638 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.801638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:32.802928 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:32.802931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:32.803475 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:32.803475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:32.803631 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:32.804250 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:29:32.804261 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:29:32.807679 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:29:32.809804 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:29:32.809811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:29:32.809814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:29:32.810124 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:29:32.810371 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:29:32.810372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:29:32.810372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:29:32.810514 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:29:32.810861 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:29:32.810868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:29:32.811040 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 14:29:32.811514 (+ 474us) client_negotiation.cc:240] Negotiation successful
0812 14:29:32.811699 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":102,"threads_started":1}
I20260812 14:29:32.812964 11292 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46211
I20260812 14:29:32.813068 11292 heartbeater.cc:464] Registering TS with master...
I20260812 14:29:32.813258 11292 heartbeater.cc:511] Master 127.23.173.254:46211 requested a full tablet report, sending...
I20260812 14:29:32.813637 10736 ts_manager.cc:194] Registered new tserver with Master: a11c06b03eaa4861a52a5a53fa6f4d58 (127.23.173.194:44057)
I20260812 14:29:32.814131 10736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:59973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:29:32.838864 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46211
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:29:32.965054 11299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:29:32.965346 11299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:29:32.965384 11299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:29:32.966934 11299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:29:32.966990 11299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:29:32.967027 11299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:29:32.969502 11299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.173.212:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:29:32.969784 11299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:29:32.970214 11299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:29:32.980592 11310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:32.980587 11307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:29:32.980587 11306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:29:32.997618 11299 server_base.cc:1034] running on GCE node
I20260812 14:29:32.997946 11299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:29:32.998237 11299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:29:32.999410 11299 hybrid_clock.cc:648] HybridClock initialized: now 1786544972999389 us; error 46 us; skew 500 ppm
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:29:33.002820 11299 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:29:33.003436 11299 webserver.cc:460] Webserver started at http://127.23.173.195:38419/ using document root <none> and password file <none>
I20260812 14:29:33.003786 11299 fs_manager.cc:359] Metadata directory not provided
I20260812 14:29:33.003912 11299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:29:33.004107 11299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:29:32 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:29:33.026381 11299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c"
format_stamp: "Formatted at 2026-08-12 14:29:33 on dist-test-slave-1l3l"
server_key: "eec4a6c08d82b064bb2eb5f453fe4064"
server_key_iv: "b9e760427437e005e7e1d04e8b7ca625"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:33.026968 11299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c"
format_stamp: "Formatted at 2026-08-12 14:29:33 on dist-test-slave-1l3l"
server_key: "eec4a6c08d82b064bb2eb5f453fe4064"
server_key_iv: "b9e760427437e005e7e1d04e8b7ca625"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:33.034505 11299 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.003s
I20260812 14:29:33.067855 11321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:33.068077 11299 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260812 14:29:33.068182 11299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c"
format_stamp: "Formatted at 2026-08-12 14:29:33 on dist-test-slave-1l3l"
server_key: "eec4a6c08d82b064bb2eb5f453fe4064"
server_key_iv: "b9e760427437e005e7e1d04e8b7ca625"
server_key_version: "kuduclusterkey@0"
I20260812 14:29:33.068305 11299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:29:33.094354 11299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:29:33.097626 11299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:29:33.097841 11299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:29:33.098114 11299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:29:33.098533 11299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:29:33.098627 11299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:33.098698 11299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:29:33.098757 11299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:29:33.108139 11299 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:42095
I20260812 14:29:33.108763 11299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:29:33.112530 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:42095 every 8 connection(s)
I20260812 14:29:33.113092 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 11299
I20260812 14:29:33.113190 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:29:33.113395 24247 external_mini_cluster.cc:1428] Setting key c4ee8ceaa7a89a4e91049fde79d46a4e
Aug 12 14:29:33 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:29:33.126590 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.112386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:46911 (local address 127.23.173.254:46211)
0812 14:29:33.112430 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.112432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.115976 (+ 3544us) server_negotiation.cc:407] Connection header received
0812 14:29:33.116659 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.116660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.116672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.116734 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:29:33.117528 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.118303 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.119807 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.120116 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.120156 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.122747 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:29:33.122749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:29:33.122749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:29:33.122757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:29:33.124413 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:33.124896 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:33.124897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:33.124898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:33.124957 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:33.125831 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:33.125832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:33.125833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:33.125981 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:29:33.126025 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.126388 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.126479 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:29:33.127353 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.111899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46211 (local address 127.23.173.195:46911)
0812 14:29:33.115830 (+ 3931us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.115884 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:33.116555 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:33.116868 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:33.116873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:33.117131 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:29:33.117410 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.117416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.118406 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.118407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.119672 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.119675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.120245 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.120246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.120386 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.120972 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:29:33.120983 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:29:33.122638 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:29:33.124517 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:29:33.124524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:29:33.124527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:29:33.124803 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:29:33.125024 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:29:33.125024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:29:33.125025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:29:33.125751 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:29:33.126070 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:29:33.126076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:29:33.126243 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 14:29:33.126948 (+ 705us) client_negotiation.cc:240] Negotiation successful
0812 14:29:33.127116 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3880,"thread_start_us":96,"threads_started":1}
I20260812 14:29:33.128014 11506 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46211
I20260812 14:29:33.128113 11506 heartbeater.cc:464] Registering TS with master...
I20260812 14:29:33.128307 11506 heartbeater.cc:511] Master 127.23.173.254:46211 requested a full tablet report, sending...
I20260812 14:29:33.128728 10736 ts_manager.cc:194] Registered new tserver with Master: e8bea7644f4c4a23a8a09b8f88eaff3c (127.23.173.195:42095)
I20260812 14:29:33.129206 10736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:46911
I20260812 14:29:33.136219 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:29:33 dist-test-slave-1l3l krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:29:33.171643 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.150942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.23.173.254:46211)
0812 14:29:33.150996 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.150999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.155370 (+ 4371us) server_negotiation.cc:407] Connection header received
0812 14:29:33.159770 (+ 4400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.159773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.159791 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.159824 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:29:33.160644 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.161367 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.162376 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.162642 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.162682 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.167806 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:29:33.167807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:29:33.167808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:29:33.167818 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:29:33.169796 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:33.170705 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:33.170706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:33.170707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:33.170780 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:29:33.170999 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:29:33.171000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:29:33.171001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:29:33.171138 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:29:33.171213 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.171435 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.171534 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:29:33.174065 10736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:29:33.176273 10736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:29:33.190623 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.179927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:42095 (local address 127.0.0.1:46744)
0812 14:29:33.181195 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.181202 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:33.181233 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:33.181658 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:33.181662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:33.181664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:29:33.181729 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.181730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.183280 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.183281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.184640 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.184643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.190291 (+ 5648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.190292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.190397 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.190415 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 14:29:33.190487 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:29:33.190539 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"thread_start_us":67,"threads_started":1}
I20260812 14:29:33.191574 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.180371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.23.173.195:42095)
0812 14:29:33.181341 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.181346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.181355 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:29:33.181392 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.181394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.181408 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.181483 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:29:33.181805 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.183201 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.185378 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.186153 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.186239 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.186297 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.191449 (+ 5152us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.191496 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":70,"threads_started":1}
I20260812 14:29:33.192430 11404 tablet_service.cc:1467] Processing CreateTablet for tablet e3a6558156c246a38c8191fa429a15d2 (DEFAULT_TABLE table=test-table [id=0772815bc6d140f28dcd5089c68296be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:29:33.192821 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6558156c246a38c8191fa429a15d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:29:33.195675 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.179687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:40565 (local address 127.0.0.1:55000)
0812 14:29:33.181002 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.181044 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:33.181105 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:33.192062 (+ 10957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:33.192064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:33.192066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:29:33.192105 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.192107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.193381 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.193382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.194445 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.194449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.195446 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.195447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.195512 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.195516 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:29:33.195564 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:29:33.195602 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":100,"threads_started":1}
I20260812 14:29:33.195914 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.179806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.23.173.193:40565)
0812 14:29:33.191835 (+ 12029us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.191841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.191853 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:29:33.191888 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.191890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.191906 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.191985 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:29:33.192172 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.193271 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.194544 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.195257 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.195327 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.195409 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.195805 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.195852 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11989,"thread_start_us":79,"threads_started":1}
I20260812 14:29:33.196615 10967 tablet_service.cc:1467] Processing CreateTablet for tablet e3a6558156c246a38c8191fa429a15d2 (DEFAULT_TABLE table=test-table [id=0772815bc6d140f28dcd5089c68296be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:29:33.196978 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6558156c246a38c8191fa429a15d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:29:33.197436 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.180567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.23.173.194:44057)
0812 14:29:33.181606 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.181616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.185077 (+ 3461us) server_negotiation.cc:407] Connection header received
0812 14:29:33.185199 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.185201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.185217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.185278 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:29:33.186452 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.187532 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.189180 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.191143 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.191232 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.191294 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.197286 (+ 5992us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.197352 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":78,"threads_started":1}
I20260812 14:29:33.197635 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.180092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:44057 (local address 127.0.0.1:59110)
0812 14:29:33.184758 (+ 4666us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.184768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:33.185109 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:33.186332 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:33.186334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:33.186336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:29:33.186376 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.186377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.187925 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.187926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.189059 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.189061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.197105 (+ 8044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.197106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.197171 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.197176 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:29:33.197518 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 14:29:33.197569 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4649,"thread_start_us":58,"threads_started":1}
I20260812 14:29:33.200423 11179 tablet_service.cc:1467] Processing CreateTablet for tablet e3a6558156c246a38c8191fa429a15d2 (DEFAULT_TABLE table=test-table [id=0772815bc6d140f28dcd5089c68296be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:29:33.200803 11179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6558156c246a38c8191fa429a15d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:29:33.204716 11545 tablet_bootstrap.cc:492] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Bootstrap starting.
I20260812 14:29:33.205766 11545 tablet_bootstrap.cc:654] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Neither blocks nor log segments found. Creating new log.
I20260812 14:29:33.206161 11545 log.cc:824] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Log is configured to *not* fsync() on all Append() calls
I20260812 14:29:33.207032 11545 tablet_bootstrap.cc:492] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: No bootstrap required, opened a new log
I20260812 14:29:33.207098 11545 ts_tablet_manager.cc:1397] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:29:33.209124 11545 raft_consensus.cc:348] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.209288 11545 raft_consensus.cc:374] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:29:33.209318 11545 raft_consensus.cc:729] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11c06b03eaa4861a52a5a53fa6f4d58, State: Initialized, Role: FOLLOWER
I20260812 14:29:33.209422 11545 consensus_queue.cc:260] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.209668 11545 ts_tablet_manager.cc:1428] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 14:29:33.209803 11292 heartbeater.cc:503] Master 127.23.173.254:46211 was elected leader, sending a full tablet report...
I20260812 14:29:33.228340 11540 tablet_bootstrap.cc:492] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Bootstrap starting.
I20260812 14:29:33.232277 11540 tablet_bootstrap.cc:654] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Neither blocks nor log segments found. Creating new log.
I20260812 14:29:33.232751 11540 log.cc:824] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:29:33.233244 11542 tablet_bootstrap.cc:492] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: Bootstrap starting.
I20260812 14:29:33.234141 11542 tablet_bootstrap.cc:654] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: Neither blocks nor log segments found. Creating new log.
I20260812 14:29:33.234508 11542 log.cc:824] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: Log is configured to *not* fsync() on all Append() calls
I20260812 14:29:33.241250 11540 tablet_bootstrap.cc:492] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: No bootstrap required, opened a new log
I20260812 14:29:33.241348 11540 ts_tablet_manager.cc:1397] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260812 14:29:33.243395 11540 raft_consensus.cc:348] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.243602 11540 raft_consensus.cc:374] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:29:33.243643 11540 raft_consensus.cc:729] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bea7644f4c4a23a8a09b8f88eaff3c, State: Initialized, Role: FOLLOWER
I20260812 14:29:33.243782 11540 consensus_queue.cc:260] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.244069 11540 ts_tablet_manager.cc:1428] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:29:33.246799 11542 tablet_bootstrap.cc:492] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: No bootstrap required, opened a new log
I20260812 14:29:33.246882 11542 ts_tablet_manager.cc:1397] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260812 14:29:33.252683 11506 heartbeater.cc:503] Master 127.23.173.254:46211 was elected leader, sending a full tablet report...
I20260812 14:29:33.254842 11542 raft_consensus.cc:348] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.255108 11542 raft_consensus.cc:374] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:29:33.255190 11542 raft_consensus.cc:729] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09feac6583247ccbd0f6d935c4ffb15, State: Initialized, Role: FOLLOWER
I20260812 14:29:33.255781 11542 consensus_queue.cc:260] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.256188 11542 ts_tablet_manager.cc:1428] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260812 14:29:33.256348 11068 heartbeater.cc:503] Master 127.23.173.254:46211 was elected leader, sending a full tablet report...
W20260812 14:29:33.284371 11070 tablet.cc:2369] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:29:33.292514 11552 raft_consensus.cc:482] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:29:33.292658 11552 raft_consensus.cc:504] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.293013 11552 leader_election.cc:288] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8bea7644f4c4a23a8a09b8f88eaff3c (127.23.173.195:42095), a11c06b03eaa4861a52a5a53fa6f4d58 (127.23.173.194:44057)
W20260812 14:29:33.299293 11293 tablet.cc:2369] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:29:33.302932 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.293663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51355 (local address 127.23.173.195:42095)
0812 14:29:33.293728 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.293730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.298942 (+ 5212us) server_negotiation.cc:407] Connection header received
0812 14:29:33.299024 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.299025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.299041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.299134 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:29:33.299883 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.300634 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.302011 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.302560 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.302608 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.302618 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.302817 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.302856 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:29:33.303401 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.293297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:42095 (local address 127.23.173.193:51355)
0812 14:29:33.298857 (+ 5560us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.298870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:33.298948 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:33.299674 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:33.299675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:33.299677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:29:33.299788 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.299791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.300752 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.300753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.301911 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.301914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.302673 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.302674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.302746 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.302750 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:29:33.303212 (+ 462us) client_negotiation.cc:240] Negotiation successful
0812 14:29:33.303258 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5534,"thread_start_us":126,"threads_started":1}
I20260812 14:29:33.304759 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.293976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:54547 (local address 127.23.173.194:44057)
0812 14:29:33.294028 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.294030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.299116 (+ 5086us) server_negotiation.cc:407] Connection header received
0812 14:29:33.299209 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.299210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.299223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.299251 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:29:33.299468 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.300280 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.303798 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.304384 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.304433 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.304441 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.304654 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.304694 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:29:33.305020 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.293806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:44057 (local address 127.23.173.193:54547)
0812 14:29:33.299063 (+ 5257us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.299070 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:29:33.299141 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:29:33.299310 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:29:33.299310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:29:33.299312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:29:33.299395 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.299397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.300402 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.300402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.301691 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:29:33.301694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.304501 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.304502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:29:33.304581 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.304585 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:29:33.304880 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 14:29:33.304921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5240,"thread_start_us":74,"threads_started":1}
I20260812 14:29:33.305260 11431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6558156c246a38c8191fa429a15d2" candidate_uuid: "b09feac6583247ccbd0f6d935c4ffb15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" is_pre_election: true
I20260812 14:29:33.305430 11431 raft_consensus.cc:2454] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09feac6583247ccbd0f6d935c4ffb15 in term 0.
I20260812 14:29:33.305639 11213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6558156c246a38c8191fa429a15d2" candidate_uuid: "b09feac6583247ccbd0f6d935c4ffb15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" is_pre_election: true
I20260812 14:29:33.305774 11213 raft_consensus.cc:2454] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09feac6583247ccbd0f6d935c4ffb15 in term 0.
I20260812 14:29:33.306008 10907 leader_election.cc:302] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a11c06b03eaa4861a52a5a53fa6f4d58, b09feac6583247ccbd0f6d935c4ffb15; no voters:
I20260812 14:29:33.306154 11552 raft_consensus.cc:2781] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:29:33.306201 11552 raft_consensus.cc:482] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:29:33.306226 11552 raft_consensus.cc:3037] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:29:33.307271 11552 raft_consensus.cc:504] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.307396 11552 leader_election.cc:288] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [CANDIDATE]: Term 1 election: Requested vote from peers e8bea7644f4c4a23a8a09b8f88eaff3c (127.23.173.195:42095), a11c06b03eaa4861a52a5a53fa6f4d58 (127.23.173.194:44057)
I20260812 14:29:33.308084 11213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6558156c246a38c8191fa429a15d2" candidate_uuid: "b09feac6583247ccbd0f6d935c4ffb15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58"
I20260812 14:29:33.308161 11213 raft_consensus.cc:3037] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:29:33.308982 11213 raft_consensus.cc:2454] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09feac6583247ccbd0f6d935c4ffb15 in term 1.
I20260812 14:29:33.309142 10907 leader_election.cc:302] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a11c06b03eaa4861a52a5a53fa6f4d58, b09feac6583247ccbd0f6d935c4ffb15; no voters:
I20260812 14:29:33.309319 11431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6558156c246a38c8191fa429a15d2" candidate_uuid: "b09feac6583247ccbd0f6d935c4ffb15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c"
I20260812 14:29:33.309406 11431 raft_consensus.cc:3037] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:29:33.310184 11431 raft_consensus.cc:2454] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09feac6583247ccbd0f6d935c4ffb15 in term 1.
I20260812 14:29:33.310448 11552 raft_consensus.cc:2781] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:29:33.310606 11552 raft_consensus.cc:686] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [term 1 LEADER]: Becoming Leader. State: Replica: b09feac6583247ccbd0f6d935c4ffb15, State: Running, Role: LEADER
I20260812 14:29:33.310693 11552 consensus_queue.cc:237] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } }
I20260812 14:29:33.311498 10736 catalog_manager.cc:5495] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09feac6583247ccbd0f6d935c4ffb15 (127.23.173.193). New cstate: current_term: 1 leader_uuid: "b09feac6583247ccbd0f6d935c4ffb15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09feac6583247ccbd0f6d935c4ffb15" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 40565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 } health_report { overall_health: UNKNOWN } } }
I20260812 14:29:33.331920 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.328845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.23.173.193:40565)
0812 14:29:33.328912 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:29:33.328914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:29:33.328923 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:29:33.328966 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:29:33.328967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:29:33.329001 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:29:33.329087 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:29:33.329232 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.329919 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.330893 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:29:33.331217 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:29:33.331291 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:29:33.331470 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:29:33.331618 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:29:33.331740 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 14:29:33.331793 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 14:29:33.331816 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:29:33.334124 11213 raft_consensus.cc:1261] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 1 FOLLOWER]: Refusing update from remote peer b09feac6583247ccbd0f6d935c4ffb15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:29:33.334209 11431 raft_consensus.cc:1261] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 1 FOLLOWER]: Refusing update from remote peer b09feac6583247ccbd0f6d935c4ffb15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:29:33.334656 11552 consensus_queue.cc:1035] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11c06b03eaa4861a52a5a53fa6f4d58" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:29:33.334828 11563 consensus_queue.cc:1035] T e3a6558156c246a38c8191fa429a15d2 P b09feac6583247ccbd0f6d935c4ffb15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bea7644f4c4a23a8a09b8f88eaff3c" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:29:33.339294 11562 mvcc.cc:205] Tried to move back new op lower bound from 7317688210773458944 to 7317688210680872960. Current Snapshot: MvccSnapshot[applied={T|T < 7317688210773458944}]
I20260812 14:29:33.339777 11568 mvcc.cc:205] Tried to move back new op lower bound from 7317688210773458944 to 7317688210680872960. Current Snapshot: MvccSnapshot[applied={T|T < 7317688210773458944}]
I20260812 14:29:33.341930 11565 mvcc.cc:205] Tried to move back new op lower bound from 7317688210773458944 to 7317688210680872960. Current Snapshot: MvccSnapshot[applied={T|T < 7317688210773458944}]
I20260812 14:29:33.342535 10736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:29:33.342633 10736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:29:33.343400 10736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 164e6946a1544db2abb2f0e10019493c: Sending DeleteTablet for 3 replicas of tablet e3a6558156c246a38c8191fa429a15d2
I20260812 14:29:33.343856 11179 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a6558156c246a38c8191fa429a15d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:29:33 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:59110
I20260812 14:29:33.343866 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a6558156c246a38c8191fa429a15d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:29:33 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:55000
I20260812 14:29:33.344106 11404 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a6558156c246a38c8191fa429a15d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:29:33 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:46744
I20260812 14:29:33.344666 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 10874
I20260812 14:29:33.352524 11585 tablet_replica.cc:330] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: stopping tablet replica
I20260812 14:29:33.352662 11585 raft_consensus.cc:2229] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:29:33.352807 11585 raft_consensus.cc:2258] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:29:33.354403 11589 tablet_replica.cc:330] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: stopping tablet replica
I20260812 14:29:33.354559 11589 raft_consensus.cc:2229] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:29:33.354723 11589 raft_consensus.cc:2258] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:29:33.358980 11585 ts_tablet_manager.cc:1905] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:29:33.362262 11585 ts_tablet_manager.cc:1918] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:29:33.362350 11585 log.cc:1196] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/e3a6558156c246a38c8191fa429a15d2
I20260812 14:29:33.362370 11589 ts_tablet_manager.cc:1905] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:29:33.362613 11585 ts_tablet_manager.cc:1939] T e3a6558156c246a38c8191fa429a15d2 P a11c06b03eaa4861a52a5a53fa6f4d58: Deleting consensus metadata
I20260812 14:29:33.363072 10713 catalog_manager.cc:4841] TS a11c06b03eaa4861a52a5a53fa6f4d58 (127.23.173.194:44057): tablet e3a6558156c246a38c8191fa429a15d2 (table test-table [id=0772815bc6d140f28dcd5089c68296be]) successfully deleted
I20260812 14:29:33.364029 11589 ts_tablet_manager.cc:1918] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:29:33.364111 11589 log.cc:1196] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/e3a6558156c246a38c8191fa429a15d2
I20260812 14:29:33.364701 11589 ts_tablet_manager.cc:1939] T e3a6558156c246a38c8191fa429a15d2 P e8bea7644f4c4a23a8a09b8f88eaff3c: Deleting consensus metadata
I20260812 14:29:33.365139 10712 catalog_manager.cc:4841] TS e8bea7644f4c4a23a8a09b8f88eaff3c (127.23.173.195:42095): tablet e3a6558156c246a38c8191fa429a15d2 (table test-table [id=0772815bc6d140f28dcd5089c68296be]) successfully deleted
W20260812 14:29:33.369242 10712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:40565 (error 108)
I20260812 14:29:33.369913 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 11077
I20260812 14:29:33.380252 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:29:33.380087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:40565 (local address 127.0.0.1:55018)
0812 14:29:33.380143 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 14:29:33.380208 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:29:33.380417 10712 catalog_manager.cc:4584] TS b09feac6583247ccbd0f6d935c4ffb15 (127.23.173.193:40565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3a6558156c246a38c8191fa429a15d2: Network error: Client connection negotiation failed: client connection to 127.23.173.193:40565: connect: Connection refused (error 111)
I20260812 14:29:33.381978 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 11299
I20260812 14:29:33.391140 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 10675
I20260812 14:29:33.408481 24247 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 14:29:33.752504 24247 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 14:29:33.752614 24247 mini_ranger.cc:66] Stopping Ranger...
I20260812 14:29:34.106853 24247 mini_ranger.cc:68] Stopped Ranger
2026-08-12 14:29:34.107 UTC [9994] LOG: received smart shutdown request
2026-08-12 14:29:34.110 UTC [9994] LOG: background worker "logical replication launcher" (PID 10006) exited with exit code 1
2026-08-12 14:29:34.111 UTC [9999] LOG: shutting down
2026-08-12 14:29:34.265 UTC [9994] LOG: database system is shut down
2026-08-12T14:29:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48974 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:29:34 dist-test-slave-1l3l krb5kdc[11605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:29:34 dist-test-slave-1l3l krb5kdc[11605](info): set up 2 sockets
Aug 12 14:29:34 dist-test-slave-1l3l krb5kdc[11605](info): commencing operation
krb5kdc: starting...
W20260812 14:29:35.541527 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:29:35 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:29:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/rangeradmin_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/rangeradmin_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/rangerlookup_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/rangerlookup_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
I20260812 14:29:35.698789 24247 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOhwVqn/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/postgres -l logfile start
2026-08-12 14:29:38.086 UTC [11648] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 14:29:38.086 UTC [11648] LOG: listening on IPv4 address "127.23.173.212", port 54273
2026-08-12 14:29:38.089 UTC [11648] LOG: listening on Unix socket "/tmp/.s.PGSQL.54273"
2026-08-12 14:29:38.109 UTC [11651] LOG: database system was shut down at 2026-08-12 14:29:36 UTC
2026-08-12 14:29:38.114 UTC [11648] LOG: database system is ready to accept connections
I20260812 14:29:39.203279 24247 mini_postgres.cc:96] Postgres bound to 54273
2026-08-12 14:29:39.210 UTC [11666] FATAL: database "slave" does not exist
127.23.173.212:54273 - accepting connections
I20260812 14:29:39.210645 24247 mini_ranger.cc:161] Starting Ranger...
I20260812 14:29:39.225354 24247 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 14:29:39.696034 24247 mini_ranger.cc:87] Created ranger Postgres database
I20260812 14:29:39.696122 24247 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-admin
2026-08-12 14:29:39,978 [I] DB FLAVOR :POSTGRES
2026-08-12 14:29:39,978 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:29:39,978 [I] Checking connection..
2026-08-12 14:29:39,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 14:29:40,242 [I] Checking connection passed.
2026-08-12 14:29:40,242 [I] --------- Verifying version history table ---------
2026-08-12 14:29:40,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:29:40,496 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:29:40,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:29:40,746 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:29:40,746 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 14:29:40,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 14:29:41.007 UTC [11780] WARNING: there is no transaction in progress
2026-08-12 14:29:41,018 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 14:29:41,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:29:41,291 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 14:29:41,291 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:29:41,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 14:29:41,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 14:29:41,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1l3l.c.gcp-upstream.internal','N') ;"
2026-08-12 14:29:42,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:29:42,320 [I] Table x_portal_user does not exist in database ranger
2026-08-12 14:29:42,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:29:42,576 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 14:29:42,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:29:42,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:29:43,089 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 14:29:43,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 14:29:44.064 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.079 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.095 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.110 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.129 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.147 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.200 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.207 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.220 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.230 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.241 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.256 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.266 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.272 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.279 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44.286 UTC [11962] WARNING: there is no transaction in progress
2026-08-12 14:29:44,940 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 14:29:44,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 14:29:45,199 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 14:29:45,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:29:45,469 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 14:29:45,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:29:45,731 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 14:29:45,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:29:45,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:29:46,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1l3l.c.gcp-upstream.internal';"
2026-08-12 14:29:46,501 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 14:29:46,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 14:29:46,767 [I] DB_PATCHES have already been applied
I20260812 14:29:46.772909 24247 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOhwVqn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 14:29:46.772976 24247 mini_ranger.cc:193] Using host: 127.23.173.212
I20260812 14:29:46.776847 24247 mini_ranger.cc:227] Ranger admin URL: http://127.23.173.212:55073
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:29:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 14:29:48.059577 24247 mini_ranger.cc:160] Time spent starting Ranger: real 8.849s user 0.002s sys 0.008s
Aug 12, 2026 2:30:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:30:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:30:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:30:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:30:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 14:30:09.891122 24247 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.173.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.173.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/rangerkms_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.173.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/rangerkms_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.173.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/HTTP_127.23.173.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 14:30:09 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545009, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 14:30:09.972709 24247 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 14:30:09.982378 24247 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 14:30:10.805318 24247 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 14:30:10.805430 24247 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms
I20260812 14:30:10.805450 24247 mini_ranger_kms.cc:227] Using postgres at 127.23.173.212:54273
2026-08-12 14:30:11,000 [I] DB FLAVOR :POSTGRES
2026-08-12 14:30:11,001 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:30:11,001 [I] Checking connection
2026-08-12 14:30:11,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 14:30:11,254 [I] connection success
2026-08-12 14:30:11,255 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 14:30:11,255 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 14:30:11,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 14:30:11,487 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 14:30:11,487 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:30:11,487 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 14:30:11,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.212:54273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 14:30:11,754 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 14:30:12.266281 24247 mini_ranger_kms.cc:310] Created kms service
I20260812 14:30:12.468920 24247 mini_ranger_kms.cc:326] Created kudu user
I20260812 14:30:12.512773 24247 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 14:30:12 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545009, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:12.917114 24247 mini_ranger_kms.cc:384] Added ranger policy
I20260812 14:30:12.917338 24247 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOhwVqn/build/release/bin/postgresql.jar:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOhwVqn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 14:30:12.917363 24247 mini_ranger_kms.cc:240] Using host: 127.23.173.212
I20260812 14:30:12.922050 24247 mini_ranger_kms.cc:276] Ranger KMS PID: 12262
I20260812 14:30:12.922129 24247 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.173.212:57973
Aug 12, 2026 2:30:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:30:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:30:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:30:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:30:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:30:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 14:30:15 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786545015, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:15.454499 24247 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.482s user 0.007s sys 0.170s
I20260812 14:30:15.454618 24247 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 14:30:15.454669 24247 mini_ranger_kms.cc:401] 127.23.173.212:57973/kms/v1/keys
Aug 12 14:30:15 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545015, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.212@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
Aug 12 14:30:16 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545015, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.212@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
Aug 12 14:30:18 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:30:18.318437 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:34061
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:34061
--ranger_config_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:18.399566 12351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:18.399808 12351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:18.399842 12351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:18.400998 12351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:18.401041 12351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:18.401060 12351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:18.401077 12351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:18.401093 12351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:18.402649 12351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.173.212:57973/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:34061
--ranger_config_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:34061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:18.402895 12351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:18.403105 12351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:18.407724 12364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:18.407938 12351 server_base.cc:1034] running on GCE node
W20260812 14:30:18.407683 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:18.407682 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:18.408303 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:18.408651 12351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:18.409798 12351 hybrid_clock.cc:648] HybridClock initialized: now 1786545018409779 us; error 38 us; skew 500 ppm
Aug 12 14:30:18 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545018, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:18.417529 12351 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:18.418004 12351 webserver.cc:460] Webserver started at http://127.23.173.254:38805/ using document root <none> and password file <none>
I20260812 14:30:18.418219 12351 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:18.418262 12351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:18.418385 12351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:30:18 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545018, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:18.463974 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "230b1ff4efb44d0eb0560e40ba1aea49"
format_stamp: "Formatted at 2026-08-12 14:30:18 on dist-test-slave-1l3l"
server_key: "715a3146120c7b03017e45e7d61815d0"
server_key_iv: "c9c57c5a1dc874d0e611ff31ed14a6d2"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:18.464428 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "230b1ff4efb44d0eb0560e40ba1aea49"
format_stamp: "Formatted at 2026-08-12 14:30:18 on dist-test-slave-1l3l"
server_key: "715a3146120c7b03017e45e7d61815d0"
server_key_iv: "c9c57c5a1dc874d0e611ff31ed14a6d2"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:18.466224 12351 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:18.477711 12373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:18.477919 12351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:18.478014 12351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "230b1ff4efb44d0eb0560e40ba1aea49"
format_stamp: "Formatted at 2026-08-12 14:30:18 on dist-test-slave-1l3l"
server_key: "715a3146120c7b03017e45e7d61815d0"
server_key_iv: "c9c57c5a1dc874d0e611ff31ed14a6d2"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:18.478102 12351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:18.489419 12351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:18.491241 12351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:18.491377 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:18.495987 12351 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:34061
I20260812 14:30:18.496022 12467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:34061 every 8 connection(s)
I20260812 14:30:18.496361 12351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:18.497150 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:18.500165 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Bootstrap starting.
I20260812 14:30:18.500881 12468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:18.501201 12468 log.cc:824] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:18.501967 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: No bootstrap required, opened a new log
I20260812 14:30:18.504413 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 12351
I20260812 14:30:18.504508 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:30:18.504665 24247 external_mini_cluster.cc:1428] Setting key 5b701b6c382651292b546fcdfc323ffa
I20260812 14:30:18.504783 12468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:18.505014 12468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:18.505070 12468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230b1ff4efb44d0eb0560e40ba1aea49, State: Initialized, Role: FOLLOWER
I20260812 14:30:18.505196 12468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:18.505280 12468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:18.505378 12468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:18.505442 12468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:18.506155 12468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:18.506263 12468 leader_election.cc:302] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230b1ff4efb44d0eb0560e40ba1aea49; no voters:
I20260812 14:30:18.506466 12468 leader_election.cc:288] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:30:18.506505 12475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:18.506744 12475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 LEADER]: Becoming Leader. State: Replica: 230b1ff4efb44d0eb0560e40ba1aea49, State: Running, Role: LEADER
I20260812 14:30:18.506791 12468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:18.507309 12475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:18.507634 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230b1ff4efb44d0eb0560e40ba1aea49. Latest consensus state: current_term: 1 leader_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } } }
I20260812 14:30:18.507692 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:18.508044 12487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:18.508560 12487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:18.508862 12468 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:30:18.510491 12506 mvcc.cc:205] Tried to move back new op lower bound from 7317688395817353216 to 7317688395806302208. Current Snapshot: MvccSnapshot[applied={T|T < 7317688395817353216}]
I20260812 14:30:18.511013 12487 catalog_manager.cc:1347] Generated new cluster ID: 959a65ad763a4faab34345de0821202d
I20260812 14:30:18.511053 12487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:30:18.512393 12478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } } }
I20260812 14:30:18.512476 12478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: This master's current role is: LEADER
Aug 12 14:30:18 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:18.521488 12487 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:30:18.521868 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:18.505138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.23.173.254:34061)
0812 14:30:18.505316 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:18.505327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:18.505522 (+ 195us) server_negotiation.cc:407] Connection header received
0812 14:30:18.505623 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:18.505628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:18.505669 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:18.505784 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:18.506476 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:18.508542 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:18.509620 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:18.510218 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:18.512056 (+ 1838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:18.517010 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:18.517022 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:18.517023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:18.517045 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:18.519805 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:18.520236 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:18.520239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:18.520240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:18.520351 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:18.520556 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:18.520557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:18.520557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:18.520831 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:18.521391 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:18.521463 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:30:18.521620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260812 14:30:18.522262 12487 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:18.525872 12487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Generated new TSK 0
I20260812 14:30:18.526083 12487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:30:19.324568 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34061
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:19.433640 12550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:19.433847 12550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:19.433878 12550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:19.435030 12550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:19.435073 12550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:19.435108 12550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:30:19.436650 12550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.173.212:57973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:19.436882 12550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:19.437126 12550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:19.442246 12561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:19.442206 12558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:19.442574 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:19.442704 12550 server_base.cc:1034] running on GCE node
I20260812 14:30:19.442955 12550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:19.443190 12550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:19.444325 12550 hybrid_clock.cc:648] HybridClock initialized: now 1786545019444314 us; error 29 us; skew 500 ppm
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:19.446833 12550 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:30:19.447222 12550 webserver.cc:460] Webserver started at http://127.23.173.193:37663/ using document root <none> and password file <none>
I20260812 14:30:19.447434 12550 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:19.447484 12550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:19.447599 12550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:19.464470 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "2f3bbe4bb8224743b95a06a6cbaeae74"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "2d1d155fcecde28e67c76ed982fccbb8"
server_key_iv: "d56ba5d0a47d35befc2ad74671edf7d8"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:19.465036 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "2f3bbe4bb8224743b95a06a6cbaeae74"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "2d1d155fcecde28e67c76ed982fccbb8"
server_key_iv: "d56ba5d0a47d35befc2ad74671edf7d8"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:19.466537 12550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:19.478659 12573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:19.478838 12550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:19.478936 12550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "2f3bbe4bb8224743b95a06a6cbaeae74"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "2d1d155fcecde28e67c76ed982fccbb8"
server_key_iv: "d56ba5d0a47d35befc2ad74671edf7d8"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:19.479018 12550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:19.524756 12550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:19.527182 12550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:19.527381 12550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:19.527740 12550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:19.528196 12550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:19.528295 12550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:19.528414 12550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:19.528501 12550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:19.535312 12550 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:39451
I20260812 14:30:19.535861 12550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:30:19.537055 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:39451 every 8 connection(s)
I20260812 14:30:19.541623 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 12550
I20260812 14:30:19.541720 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:30:19.541882 24247 external_mini_cluster.cc:1428] Setting key 07373f75e4e7c8a44ded44f3a8d6e192
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:19.555291 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:19.537798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34061 (local address 127.23.173.193:39297)
0812 14:30:19.539329 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:19.539378 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:19.540692 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:19.541144 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:19.541149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:19.541387 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:19.545521 (+ 4134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:19.545530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.546507 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.546508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:19.547747 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:19.547751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.548279 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.548280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:19.548426 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:19.549003 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:19.549014 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:19.550604 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:19.553702 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:19.553710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:19.553712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:19.553978 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:19.554181 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:19.554182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:19.554182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:19.554303 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:19.554671 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:19.554677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:19.554831 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 14:30:19.554924 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 14:30:19.555070 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"thread_start_us":98,"threads_started":1}
I20260812 14:30:19.555881 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:19.537820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:39297 (local address 127.23.173.254:34061)
0812 14:30:19.540897 (+ 3077us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:19.540899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:19.540908 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:19.540936 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:19.540937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:19.540952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:19.541012 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:19.545640 (+ 4628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.546390 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.547839 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.548135 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.548196 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:19.550708 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:19.550709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:19.550710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:19.550717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:19.553615 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:19.554060 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:19.554061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:19.554062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:19.554125 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:19.554366 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:19.554366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:19.554367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:19.554585 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:19.554643 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:19.555680 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0812 14:30:19.555794 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3050,"thread_start_us":53,"threads_started":1}
I20260812 14:30:19.556631 12775 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34061
I20260812 14:30:19.556737 12775 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:19.556945 12775 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:19.557610 12398 ts_manager.cc:194] Registered new tserver with Master: 2f3bbe4bb8224743b95a06a6cbaeae74 (127.23.173.193:39451)
I20260812 14:30:19.558547 12398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:39297
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:30:19.586601 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34061
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:19.705965 12781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:19.706326 12781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:19.706439 12781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:19.708246 12781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:19.708475 12781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:19.708590 12781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:30:19.710793 12781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.173.212:57973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:19.711264 12781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:19.711578 12781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:19.718001 12805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:19.718215 12800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:19.718037 12793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:19.718423 12781 server_base.cc:1034] running on GCE node
I20260812 14:30:19.718638 12781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:19.718926 12781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:19.720098 12781 hybrid_clock.cc:648] HybridClock initialized: now 1786545019720073 us; error 35 us; skew 500 ppm
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:19.723443 12781 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:30:19.724041 12781 webserver.cc:460] Webserver started at http://127.23.173.194:33099/ using document root <none> and password file <none>
I20260812 14:30:19.724336 12781 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:19.724408 12781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:19.724592 12781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:19.746258 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "a6b5119361bb400d8b75cabc43b6c121"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "35a2f6adda99951c764fd030887a7167"
server_key_iv: "0d654563249ee0c855fc78d3e5457329"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:19.746754 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "a6b5119361bb400d8b75cabc43b6c121"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "35a2f6adda99951c764fd030887a7167"
server_key_iv: "0d654563249ee0c855fc78d3e5457329"
server_key_version: "kuduclusterkey@0"
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:19.752799 12781 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260812 14:30:19.773658 12816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:19.782611 12781 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260812 14:30:19.782727 12781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "a6b5119361bb400d8b75cabc43b6c121"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "35a2f6adda99951c764fd030887a7167"
server_key_iv: "0d654563249ee0c855fc78d3e5457329"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:19.782819 12781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:19.799111 12781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:19.801592 12781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:19.801798 12781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:19.802076 12781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:19.802508 12781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:19.802608 12781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:19.802683 12781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:19.802742 12781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:19.809463 12781 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:33229
I20260812 14:30:19.809973 12781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:30:19.810464 13008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:33229 every 8 connection(s)
I20260812 14:30:19.815412 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 12781
I20260812 14:30:19.815511 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:30:19.815742 24247 external_mini_cluster.cc:1428] Setting key 1f88dc87f0b3bf365c65fa1aa2505b4d
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:19.820724 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:19.811060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60881 (local address 127.23.173.254:34061)
0812 14:30:19.811109 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:19.811111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:19.811355 (+ 244us) server_negotiation.cc:407] Connection header received
0812 14:30:19.812491 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:19.812492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:19.812518 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:19.812586 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:19.813280 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.813790 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.815198 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.815428 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.815487 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:19.817725 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:19.817726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:19.817727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:19.817734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:19.819186 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:19.819645 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:19.819646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:19.819646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:19.819718 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:19.820019 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:19.820019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:19.820020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:19.820157 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:19.820197 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:19.820549 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 14:30:19.820640 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:30:19.821208 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:19.811046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34061 (local address 127.23.173.194:60881)
0812 14:30:19.811220 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:19.811263 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:19.812242 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:19.812762 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:19.812767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:19.812959 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:19.813164 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:19.813168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.813880 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.813881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:19.815106 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:19.815108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:19.815574 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:19.815575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:19.815693 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:19.816241 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:19.816251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:19.817637 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:19.819276 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:19.819283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:19.819285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:19.819561 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:19.819795 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:19.819796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:19.819797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:19.819926 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:19.820242 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:19.820249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:19.820402 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 14:30:19.820820 (+ 418us) client_negotiation.cc:240] Negotiation successful
0812 14:30:19.820992 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260812 14:30:19.821743 13017 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34061
I20260812 14:30:19.821842 13017 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:19.822018 13017 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:19.822400 12398 ts_manager.cc:194] Registered new tserver with Master: a6b5119361bb400d8b75cabc43b6c121 (127.23.173.194:33229)
I20260812 14:30:19.822863 12398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:60881
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:30:19.846596 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34061
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:19.972504 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:19.972843 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:19.972936 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:19.974685 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:19.974783 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:19.974869 13027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:30:19.977001 13027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.173.212:57973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:19.977509 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:19.977797 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:19.984334 13044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:19.984720 13039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:19.985729 13027 server_base.cc:1034] running on GCE node
W20260812 14:30:19.986222 13038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:19.986529 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:19.986794 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:19.988492 13027 hybrid_clock.cc:648] HybridClock initialized: now 1786545019988471 us; error 34 us; skew 500 ppm
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:19.991743 13027 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:30:19.992223 13027 webserver.cc:460] Webserver started at http://127.23.173.195:46003/ using document root <none> and password file <none>
I20260812 14:30:19.992466 13027 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:19.992514 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:19.992628 13027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:30:19 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:20.019960 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "46a2c352a4764a10add81c8a4c54d91e"
format_stamp: "Formatted at 2026-08-12 14:30:20 on dist-test-slave-1l3l"
server_key: "c8739b22869fe8d217acc1d83e5e501e"
server_key_iv: "b7b9985a134d4623c3154ebbc9b2490d"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.020505 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "46a2c352a4764a10add81c8a4c54d91e"
format_stamp: "Formatted at 2026-08-12 14:30:20 on dist-test-slave-1l3l"
server_key: "c8739b22869fe8d217acc1d83e5e501e"
server_key_iv: "b7b9985a134d4623c3154ebbc9b2490d"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.023635 13027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260812 14:30:20.037328 13055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.037595 13027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:20.037725 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "46a2c352a4764a10add81c8a4c54d91e"
format_stamp: "Formatted at 2026-08-12 14:30:20 on dist-test-slave-1l3l"
server_key: "c8739b22869fe8d217acc1d83e5e501e"
server_key_iv: "b7b9985a134d4623c3154ebbc9b2490d"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.037819 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:20.059520 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:20.062069 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:20.062266 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:20.062561 13027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:20.062958 13027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:20.063045 13027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.063112 13027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:20.063165 13027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.069733 13027 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:33363
I20260812 14:30:20.070278 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:30:20.076112 13253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:33363 every 8 connection(s)
I20260812 14:30:20.080173 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 13027
I20260812 14:30:20.080297 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:30:20.080495 24247 external_mini_cluster.cc:1428] Setting key e259b108acb5c2f83d86ebf214747a34
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:20.091428 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.072313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:48737 (local address 127.23.173.254:34061)
0812 14:30:20.072358 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:20.072360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:20.080021 (+ 7661us) server_negotiation.cc:407] Connection header received
0812 14:30:20.081626 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:20.081627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:20.081640 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:20.081671 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:20.082567 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.083357 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.084930 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.085239 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.085280 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.088059 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:20.088060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:20.088061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:20.088069 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:20.089797 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.090302 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.090304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.090304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.090367 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.090664 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.090665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.090665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.090802 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:20.090857 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:20.091244 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 14:30:20.091335 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:30:20.092707 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.072024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34061 (local address 127.23.173.195:48737)
0812 14:30:20.079838 (+ 7814us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:20.079908 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:20.081475 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:20.081883 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:20.081888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:20.082155 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:20.082450 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.082457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.083477 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.083478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.084809 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.084813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.085387 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.085388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.085528 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.086145 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:20.086155 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:20.087941 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:20.089905 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.089911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.089914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.090204 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.090441 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.090442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.090442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.090577 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.090906 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:20.090913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:20.091078 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 14:30:20.092047 (+ 969us) client_negotiation.cc:240] Negotiation successful
0812 14:30:20.092467 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7755,"thread_start_us":88,"threads_started":1}
I20260812 14:30:20.093359 13254 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34061
I20260812 14:30:20.093456 13254 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.093652 13254 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:20.094164 12398 ts_manager.cc:194] Registered new tserver with Master: 46a2c352a4764a10add81c8a4c54d91e (127.23.173.195:33363)
I20260812 14:30:20.094636 12398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:48737
I20260812 14:30:20.103067 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:30:20.103958 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 12550
I20260812 14:30:20.124470 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 12781
I20260812 14:30:20.133921 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 13027
I20260812 14:30:20.143909 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 12351
I20260812 14:30:20.164590 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:34061
--webserver_interface=127.23.173.254
--webserver_port=38805
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:34061
--ranger_config_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:20.247282 13258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:20.247479 13258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:20.247500 13258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:20.248667 13258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:20.248708 13258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:20.248721 13258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:20.248732 13258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:20.248746 13258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:20.250353 13258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:34061
--ranger_config_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:34061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:20.250587 13258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:20.250782 13258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:20.255298 13266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:20.255298 13271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:20.255316 13268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:20.255589 13258 server_base.cc:1034] running on GCE node
I20260812 14:30:20.255849 13258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:20.256129 13258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:20.257263 13258 hybrid_clock.cc:648] HybridClock initialized: now 1786545020257251 us; error 31 us; skew 500 ppm
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:20.259684 13258 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:20.260099 13258 webserver.cc:460] Webserver started at http://127.23.173.254:38805/ using document root <none> and password file <none>
I20260812 14:30:20.260282 13258 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:20.260331 13258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:20.298532 13258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:20.299345 13280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.299531 13258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:20.299623 13258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "230b1ff4efb44d0eb0560e40ba1aea49"
format_stamp: "Formatted at 2026-08-12 14:30:18 on dist-test-slave-1l3l"
server_key: "715a3146120c7b03017e45e7d61815d0"
server_key_iv: "c9c57c5a1dc874d0e611ff31ed14a6d2"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.300058 13258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:20.320981 13258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:20.322846 13258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:20.322990 13258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:20.326874 13258 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:34061
I20260812 14:30:20.326920 13376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:34061 every 8 connection(s)
I20260812 14:30:20.327265 13258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:20.328626 13377 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:30:20.329257 13377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Bootstrap starting.
I20260812 14:30:20.330266 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 13258
I20260812 14:30:20.330531 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:39451
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:34061
--webserver_port=37663
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:20.337142 13377 log.cc:824] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:20.338737 13377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:30:20.338981 13377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Bootstrap complete.
I20260812 14:30:20.341150 13377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:20.341387 13377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230b1ff4efb44d0eb0560e40ba1aea49, State: Initialized, Role: FOLLOWER
I20260812 14:30:20.341545 13377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:20.341619 13377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:20.341666 13377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:20.341710 13377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:30:20.342682 13377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:20.342782 13377 leader_election.cc:302] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230b1ff4efb44d0eb0560e40ba1aea49; no voters:
I20260812 14:30:20.343025 13377 leader_election.cc:288] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:30:20.343073 13381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:30:20.343186 13381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [term 2 LEADER]: Becoming Leader. State: Replica: 230b1ff4efb44d0eb0560e40ba1aea49, State: Running, Role: LEADER
I20260812 14:30:20.343329 13377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:20.343367 13381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } }
I20260812 14:30:20.343755 13386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } } }
I20260812 14:30:20.343861 13386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:20.343752 13384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230b1ff4efb44d0eb0560e40ba1aea49. Latest consensus state: current_term: 2 leader_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230b1ff4efb44d0eb0560e40ba1aea49" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 34061 } } }
I20260812 14:30:20.343900 13384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:20.347191 13377 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:30:20.349210 13389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:20.350087 13389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:20.350363 13389 catalog_manager.cc:1259] Loaded cluster ID: 959a65ad763a4faab34345de0821202d
I20260812 14:30:20.350467 13389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:30:20.352243 13389 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:20.352499 13389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Loaded TSK: 0
I20260812 14:30:20.352780 13389 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:30:20.418562 13379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:20.418744 13379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:20.418776 13379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:20.420115 13379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:20.420174 13379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:20.420212 13379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:30:20.422281 13379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:39451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=37663
--tserver_master_addrs=127.23.173.254:34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:20.422518 13379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:20.422818 13379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:20.428514 13440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:20.428609 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:20.428508 13437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:20.429977 13379 server_base.cc:1034] running on GCE node
I20260812 14:30:20.430198 13379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:20.430466 13379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:20.431630 13379 hybrid_clock.cc:648] HybridClock initialized: now 1786545020431610 us; error 38 us; skew 500 ppm
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:20.434165 13379 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:30:20.434626 13379 webserver.cc:460] Webserver started at http://127.23.173.193:37663/ using document root <none> and password file <none>
I20260812 14:30:20.434859 13379 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:20.434934 13379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:20.458256 13379 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260812 14:30:20.459803 13459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.460003 13379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:20.460089 13379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "2f3bbe4bb8224743b95a06a6cbaeae74"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "2d1d155fcecde28e67c76ed982fccbb8"
server_key_iv: "d56ba5d0a47d35befc2ad74671edf7d8"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.461297 13379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:20.480057 13379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:20.482010 13379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:20.482198 13379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:20.482481 13379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:20.482842 13379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:20.482882 13379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.482911 13379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:20.482954 13379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.489641 13379 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:39451
I20260812 14:30:20.490119 13379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:30:20.491736 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:39451 every 8 connection(s)
I20260812 14:30:20.498634 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 13379
I20260812 14:30:20.498939 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:33229
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:34061
--webserver_port=33099
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:20.506119 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.492822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34061 (local address 127.23.173.193:35753)
0812 14:30:20.495930 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:20.495991 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:20.496595 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:20.497066 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:20.497069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:20.497300 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:20.497550 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.497555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.498711 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.498712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.499874 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.499878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.500771 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.500772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.500916 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.501407 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:20.501417 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:20.502599 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:20.504434 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.504442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.504444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.504720 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.504995 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.504996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.504996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.505117 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.505561 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:20.505567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:20.505713 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 14:30:20.505813 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 14:30:20.505955 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":2830,"threads_started":1}
I20260812 14:30:20.506745 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.493120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:35753 (local address 127.23.173.254:34061)
0812 14:30:20.493438 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:20.493449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:20.496729 (+ 3280us) server_negotiation.cc:407] Connection header received
0812 14:30:20.496848 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:20.496852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:20.496893 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:20.497009 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:20.497658 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.498586 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.499977 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.500544 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.500655 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.502740 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:20.502743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:20.502743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:20.502752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:20.504346 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.504809 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.504812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.504813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.504926 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.505181 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.505182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.505182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.505456 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:20.505532 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:20.506504 (+ 972us) server_negotiation.cc:299] Negotiation successful
0812 14:30:20.506594 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260812 14:30:20.509241 13646 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34061
I20260812 14:30:20.509333 13646 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.509514 13646 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
W20260812 14:30:20.510389 13646 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:34061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:30:20.510466 13646 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.510527 13646 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:20.510937 13646 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.511015 13646 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
W20260812 14:30:20.511389 13646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:30:20.605279 13656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:20.605620 13656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:20.605713 13656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:20.607496 13656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:20.607615 13656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:20.607695 13656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:30:20.609897 13656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:33229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=33099
--tserver_master_addrs=127.23.173.254:34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:20.610363 13656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:20.610703 13656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:20.616339 13674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:20.616391 13665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:20.616884 13656 server_base.cc:1034] running on GCE node
W20260812 14:30:20.617079 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:20.617324 13656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:20.617604 13656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:20.618780 13656 hybrid_clock.cc:648] HybridClock initialized: now 1786545020618759 us; error 38 us; skew 500 ppm
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:20.622071 13656 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:30:20.622618 13656 webserver.cc:460] Webserver started at http://127.23.173.194:33099/ using document root <none> and password file <none>
I20260812 14:30:20.622893 13656 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:20.622979 13656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:20.655784 13656 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260812 14:30:20.657078 13684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.657311 13656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:30:20.657413 13656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "a6b5119361bb400d8b75cabc43b6c121"
format_stamp: "Formatted at 2026-08-12 14:30:19 on dist-test-slave-1l3l"
server_key: "35a2f6adda99951c764fd030887a7167"
server_key_iv: "0d654563249ee0c855fc78d3e5457329"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.657779 13656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:20.682469 13656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:20.684368 13656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:20.684513 13656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:20.684738 13656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:20.685088 13656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:20.685125 13656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.685154 13656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:20.685178 13656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.691932 13656 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:33229
I20260812 14:30:20.692305 13656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:30:20.692727 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:33229 every 8 connection(s)
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:20.701104 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 13656
I20260812 14:30:20.701391 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.212:57973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:33363
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:34061
--webserver_port=46003
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:20.703315 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.693076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:43751 (local address 127.23.173.254:34061)
0812 14:30:20.693146 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:20.693148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:20.693350 (+ 202us) server_negotiation.cc:407] Connection header received
0812 14:30:20.693812 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:20.693814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:20.693830 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:20.693915 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:20.694446 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.695200 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.696855 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.697154 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.697219 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.699880 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:20.699881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:20.699881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:20.699889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:20.701571 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.702088 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.702090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.702090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.702153 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.702500 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.702501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.702501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.702645 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:20.702696 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:20.703105 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 14:30:20.703197 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:30:20.703868 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.693019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34061 (local address 127.23.173.194:43751)
0812 14:30:20.693240 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:20.693271 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:20.693708 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:20.693969 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:20.693972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:20.694127 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:20.694311 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.694315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.695310 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.695311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.696728 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.696732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.697263 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.697265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.697438 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.698130 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:20.698140 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:20.699761 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:20.701672 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.701679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.701681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.701992 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.702269 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.702270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.702271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.702410 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.702748 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:20.702754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:20.702931 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 14:30:20.703451 (+ 520us) client_negotiation.cc:240] Negotiation successful
0812 14:30:20.703615 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260812 14:30:20.704396 13867 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34061
I20260812 14:30:20.704504 13867 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.704701 13867 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
W20260812 14:30:20.705343 13867 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:34061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:30:20.705411 13867 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.705472 13867 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:20.705832 13867 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.705919 13867 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
W20260812 14:30:20.706249 13867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:30:20.803737 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:20.803979 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:20.804011 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:20.805231 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:20.805281 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:20.805316 13871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:30:20.806929 13871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.212:57973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:33363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=46003
--tserver_master_addrs=127.23.173.254:34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:20.807322 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:20.807587 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:20.812834 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:20.813020 13883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:20.813491 13871 server_base.cc:1034] running on GCE node
W20260812 14:30:20.815822 13880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:20.816052 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:20.816294 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:20.817474 13871 hybrid_clock.cc:648] HybridClock initialized: now 1786545020817453 us; error 35 us; skew 500 ppm
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:20.820356 13871 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:30:20.820822 13871 webserver.cc:460] Webserver started at http://127.23.173.195:46003/ using document root <none> and password file <none>
I20260812 14:30:20.821070 13871 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:20.821139 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:20.853017 13871 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.001s sys 0.000s
I20260812 14:30:20.854462 13898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.854672 13871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:30:20.854779 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "46a2c352a4764a10add81c8a4c54d91e"
format_stamp: "Formatted at 2026-08-12 14:30:20 on dist-test-slave-1l3l"
server_key: "c8739b22869fe8d217acc1d83e5e501e"
server_key_iv: "b7b9985a134d4623c3154ebbc9b2490d"
server_key_version: "kuduclusterkey@0"
I20260812 14:30:20.855289 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:20.868417 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:20.870450 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:20.870617 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:20.870885 13871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:20.871269 13871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:20.871322 13871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.871358 13871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:20.871394 13871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:20.878002 13871 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:33363
I20260812 14:30:20.878376 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:30:20.879268 14100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:33363 every 8 connection(s)
I20260812 14:30:20.882761 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 13871
Aug 12 14:30:20 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:20.888352 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.879170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60861 (local address 127.23.173.254:34061)
0812 14:30:20.879217 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:20.879219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:20.879576 (+ 357us) server_negotiation.cc:407] Connection header received
0812 14:30:20.880100 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:20.880102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:20.880115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:20.880184 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:20.880698 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.881439 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.882421 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.882742 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.882824 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.885211 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:20.885213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:20.885213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:20.885222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:20.886867 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.887302 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.887303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.887303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.887363 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.887594 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.887595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.887595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.887732 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:20.887792 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:20.888162 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 14:30:20.888246 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:30:20.888852 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.879178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:34061 (local address 127.23.173.195:60861)
0812 14:30:20.879429 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:20.879474 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:20.879987 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:20.880251 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:20.880254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:20.880419 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:20.880606 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.880610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.881540 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.881541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.882330 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:20.882332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.882922 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.882923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:20.883043 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.883634 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:20.883644 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:20.885114 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:20.886969 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.886976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.886978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.887226 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.887419 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:20.887420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:20.887420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:20.887534 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:20.887794 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:20.887801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:20.888011 (+ 210us) client_negotiation.cc:769] Sending connection context
0812 14:30:20.888129 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 14:30:20.888298 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
I20260812 14:30:20.889357 14102 heartbeater.cc:347] Connected to a master server at 127.23.173.254:34061
I20260812 14:30:20.889464 14102 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.889663 14102 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
W20260812 14:30:20.890338 14102 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:34061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:30:20.890404 14102 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.890472 14102 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:20.890844 14102 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:20.890939 14102 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
W20260812 14:30:20.891299 14102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:30:20.891381 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:20.883264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.23.173.254:34061)
0812 14:30:20.883603 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:20.883604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:20.883611 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:20.883631 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:20.883631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:20.883642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:20.883737 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:20.884529 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.885080 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.886076 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:20.886322 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:20.886407 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:20.887821 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:20.887822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:20.887822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:20.887827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:20.889512 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.889948 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.889950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.889950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.890017 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:20.890538 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:20.890539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:20.890539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:20.890666 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:20.891056 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:20.891107 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:30:20.891186 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":68,"threads_started":1}
Aug 12 14:30:21 dist-test-slave-1l3l krb5kdc[11605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786545021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:21.511560 13646 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:21.511729 13646 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:21.512094 13312 ts_manager.cc:194] Registered new tserver with Master: 2f3bbe4bb8224743b95a06a6cbaeae74 (127.23.173.193:39451)
I20260812 14:30:21.512799 13312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:35753
I20260812 14:30:21.706429 13867 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:21.706602 13867 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:21.706883 13312 ts_manager.cc:194] Registered new tserver with Master: a6b5119361bb400d8b75cabc43b6c121 (127.23.173.194:33229)
I20260812 14:30:21.707281 13312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:43751
Aug 12 14:30:21 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
I20260812 14:30:21.891467 14102 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:21.891620 14102 heartbeater.cc:511] Master 127.23.173.254:34061 requested a full tablet report, sending...
I20260812 14:30:21.892004 13312 ts_manager.cc:194] Registered new tserver with Master: 46a2c352a4764a10add81c8a4c54d91e (127.23.173.195:33363)
I20260812 14:30:21.892529 13312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:60861
I20260812 14:30:21.893967 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:30:21.902913 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.894988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.23.173.254:34061)
0812 14:30:21.895249 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:21.895251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:21.895418 (+ 167us) server_negotiation.cc:407] Connection header received
0812 14:30:21.895439 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:21.895440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:21.895452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:21.895505 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:21.896185 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.896896 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.897842 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.898144 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.898200 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.899618 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:21.899619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:21.899620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:21.899627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:21.901354 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:21.901711 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:21.901712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:21.901712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:21.901775 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:21.901956 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:21.901956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:21.901957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:21.902100 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:21.902246 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:21.902685 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 14:30:21.902778 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":73,"threads_started":1}
I20260812 14:30:21.905292 13312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:30:21.907604 13312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:30:21.917039 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.911117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:33363 (local address 127.0.0.1:50608)
0812 14:30:21.912089 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:21.912130 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:21.912187 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:21.912608 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:21.912611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:21.912613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:21.912671 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:21.912673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.914215 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.914216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:21.915450 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:21.915453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.916754 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.916755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:21.916834 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.916846 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:30:21.916906 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:30:21.916952 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":69,"threads_started":1}
I20260812 14:30:21.917572 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.911650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.23.173.195:33363)
0812 14:30:21.912402 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:21.912407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:21.912416 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:21.912447 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:21.912449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:21.912463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:21.912525 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:21.912736 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.914145 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.915545 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.916598 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.916664 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.916719 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:21.917452 (+ 733us) server_negotiation.cc:299] Negotiation successful
0812 14:30:21.917496 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":69,"threads_started":1}
I20260812 14:30:21.918378 13974 tablet_service.cc:1467] Processing CreateTablet for tablet ba20f82abaa64a20bc3bf4822c6e82c8 (DEFAULT_TABLE table=test-table [id=6fca00baf2f741a8bdb9423594b1a4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:21.918817 13974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20f82abaa64a20bc3bf4822c6e82c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:21.922487 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.911456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:33229 (local address 127.0.0.1:37668)
0812 14:30:21.913903 (+ 2447us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:21.913911 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:21.913948 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:21.918296 (+ 4348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:21.918297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:21.918299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:21.918336 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:21.918337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.919885 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.919886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:21.920890 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:21.920893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.921948 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.921949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:21.922323 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.922329 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:30:21.922376 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:30:21.922416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2432,"thread_start_us":64,"threads_started":1}
I20260812 14:30:21.922961 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.911579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.23.173.194:33229)
0812 14:30:21.918057 (+ 6478us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:21.918061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:21.918072 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:30:21.918105 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:21.918107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:21.918123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:21.918200 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:21.918410 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.919786 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.921001 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.921790 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.921859 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.921911 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:21.922835 (+ 924us) server_negotiation.cc:299] Negotiation successful
0812 14:30:21.922881 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6431,"thread_start_us":51,"threads_started":1}
I20260812 14:30:21.923471 14165 tablet_bootstrap.cc:492] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Bootstrap starting.
I20260812 14:30:21.923791 13751 tablet_service.cc:1467] Processing CreateTablet for tablet ba20f82abaa64a20bc3bf4822c6e82c8 (DEFAULT_TABLE table=test-table [id=6fca00baf2f741a8bdb9423594b1a4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:21.924154 13751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20f82abaa64a20bc3bf4822c6e82c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:21.924206 14165 tablet_bootstrap.cc:654] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:21.924500 14165 log.cc:824] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:21.925460 14165 tablet_bootstrap.cc:492] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: No bootstrap required, opened a new log
I20260812 14:30:21.925519 14165 ts_tablet_manager.cc:1397] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:30:21.927850 14167 tablet_bootstrap.cc:492] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Bootstrap starting.
I20260812 14:30:21.928287 14165 raft_consensus.cc:348] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:21.928516 14167 tablet_bootstrap.cc:654] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:21.928536 14165 raft_consensus.cc:374] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:21.928580 14165 raft_consensus.cc:729] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a2c352a4764a10add81c8a4c54d91e, State: Initialized, Role: FOLLOWER
I20260812 14:30:21.928727 14165 consensus_queue.cc:260] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:21.928953 14167 log.cc:824] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:21.929164 14165 ts_tablet_manager.cc:1428] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 14:30:21.929317 14102 heartbeater.cc:503] Master 127.23.173.254:34061 was elected leader, sending a full tablet report...
I20260812 14:30:21.932507 14167 tablet_bootstrap.cc:492] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: No bootstrap required, opened a new log
I20260812 14:30:21.932651 14167 ts_tablet_manager.cc:1397] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 14:30:21.933254 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.911309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39451 (local address 127.0.0.1:54304)
0812 14:30:21.912275 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:21.912282 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:21.912312 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:21.929495 (+ 17183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:21.929496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:21.929499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:21.929553 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:21.929555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.930884 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.930885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:21.932051 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:21.932054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.933024 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.933025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:21.933100 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.933105 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:30:21.933133 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 14:30:21.933175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":49,"threads_started":1}
I20260812 14:30:21.933784 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:21.911447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.23.173.193:39451)
0812 14:30:21.929298 (+ 17851us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:21.929302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:21.929315 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:30:21.929350 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:21.929352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:21.929368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:21.929636 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:21.929725 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.930779 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.932155 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:21.932917 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:21.933563 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:21.933635 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:21.933661 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 14:30:21.933703 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17809,"thread_start_us":62,"threads_started":1}
I20260812 14:30:21.934525 13538 tablet_service.cc:1467] Processing CreateTablet for tablet ba20f82abaa64a20bc3bf4822c6e82c8 (DEFAULT_TABLE table=test-table [id=6fca00baf2f741a8bdb9423594b1a4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:21.934859 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20f82abaa64a20bc3bf4822c6e82c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:21.935210 14167 raft_consensus.cc:348] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:21.935391 14167 raft_consensus.cc:374] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:21.935427 14167 raft_consensus.cc:729] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b5119361bb400d8b75cabc43b6c121, State: Initialized, Role: FOLLOWER
I20260812 14:30:21.935549 14167 consensus_queue.cc:260] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:21.935842 14167 ts_tablet_manager.cc:1428] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:30:21.935951 13867 heartbeater.cc:503] Master 127.23.173.254:34061 was elected leader, sending a full tablet report...
I20260812 14:30:21.938289 14175 tablet_bootstrap.cc:492] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: Bootstrap starting.
I20260812 14:30:21.938964 14175 tablet_bootstrap.cc:654] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:21.939265 14175 log.cc:824] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:21.940104 14175 tablet_bootstrap.cc:492] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: No bootstrap required, opened a new log
I20260812 14:30:21.940171 14175 ts_tablet_manager.cc:1397] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:30:21.942510 14175 raft_consensus.cc:348] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:21.942675 14175 raft_consensus.cc:374] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:21.942708 14175 raft_consensus.cc:729] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3bbe4bb8224743b95a06a6cbaeae74, State: Initialized, Role: FOLLOWER
I20260812 14:30:21.942867 14175 consensus_queue.cc:260] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:21.943141 14175 ts_tablet_manager.cc:1428] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:30:21.943171 13646 heartbeater.cc:503] Master 127.23.173.254:34061 was elected leader, sending a full tablet report...
W20260812 14:30:21.943482 13868 tablet.cc:2369] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:30:21.993249 13651 tablet.cc:2369] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:30:22 dist-test-slave-1l3l krb5kdc[11605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for HTTP/127.23.173.212@KRBTEST.COM
W20260812 14:30:22.130057 14103 tablet.cc:2369] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:30:22.132522 14174 raft_consensus.cc:482] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:22.132673 14174 raft_consensus.cc:504] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:22.133119 14174 leader_election.cc:288] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46a2c352a4764a10add81c8a4c54d91e (127.23.173.195:33363), 2f3bbe4bb8224743b95a06a6cbaeae74 (127.23.173.193:39451)
I20260812 14:30:22.137836 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:22.133794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:42999 (local address 127.23.173.195:33363)
0812 14:30:22.133905 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:22.133907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:22.134138 (+ 231us) server_negotiation.cc:407] Connection header received
0812 14:30:22.134238 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:22.134239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:22.134249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:22.134278 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:22.134522 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.135261 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.136829 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.137411 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.137464 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:22.137474 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:22.137716 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 14:30:22.137762 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 14:30:22.138305 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:22.133347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:33363 (local address 127.23.173.194:42999)
0812 14:30:22.134069 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:22.134081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:22.134166 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:22.134346 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:22.134347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:22.134349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:22.134445 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:22.134447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.135393 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.135394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:22.136714 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:22.136718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.137556 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.137557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:22.137633 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:22.137638 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:30:22.138150 (+ 512us) client_negotiation.cc:240] Negotiation successful
0812 14:30:22.138202 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":111,"threads_started":1}
I20260812 14:30:22.138801 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20f82abaa64a20bc3bf4822c6e82c8" candidate_uuid: "a6b5119361bb400d8b75cabc43b6c121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a2c352a4764a10add81c8a4c54d91e" is_pre_election: true
I20260812 14:30:22.138993 14012 raft_consensus.cc:2454] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b5119361bb400d8b75cabc43b6c121 in term 0.
I20260812 14:30:22.139246 13692 leader_election.cc:302] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a2c352a4764a10add81c8a4c54d91e, a6b5119361bb400d8b75cabc43b6c121; no voters:
I20260812 14:30:22.139385 14174 raft_consensus.cc:2781] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:30:22.139428 14174 raft_consensus.cc:482] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:30:22.139448 14174 raft_consensus.cc:3037] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:22.140443 14174 raft_consensus.cc:504] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:22.140585 14174 leader_election.cc:288] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [CANDIDATE]: Term 1 election: Requested vote from peers 46a2c352a4764a10add81c8a4c54d91e (127.23.173.195:33363), 2f3bbe4bb8224743b95a06a6cbaeae74 (127.23.173.193:39451)
I20260812 14:30:22.140761 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20f82abaa64a20bc3bf4822c6e82c8" candidate_uuid: "a6b5119361bb400d8b75cabc43b6c121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a2c352a4764a10add81c8a4c54d91e"
I20260812 14:30:22.140869 14012 raft_consensus.cc:3037] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:22.141813 14012 raft_consensus.cc:2454] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b5119361bb400d8b75cabc43b6c121 in term 1.
I20260812 14:30:22.142050 13692 leader_election.cc:302] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a2c352a4764a10add81c8a4c54d91e, a6b5119361bb400d8b75cabc43b6c121; no voters:
I20260812 14:30:22.142146 14174 raft_consensus.cc:2781] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:22.142330 14174 raft_consensus.cc:686] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 1 LEADER]: Becoming Leader. State: Replica: a6b5119361bb400d8b75cabc43b6c121, State: Running, Role: LEADER
I20260812 14:30:22.142427 14174 consensus_queue.cc:237] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } }
I20260812 14:30:22.143273 13312 catalog_manager.cc:5495] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6b5119361bb400d8b75cabc43b6c121 (127.23.173.194). New cstate: current_term: 1 leader_uuid: "a6b5119361bb400d8b75cabc43b6c121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6b5119361bb400d8b75cabc43b6c121" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 33229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 } health_report { overall_health: UNKNOWN } } }
I20260812 14:30:22.145831 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:22.133856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:53961 (local address 127.23.173.193:39451)
0812 14:30:22.133941 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:22.133942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:22.138734 (+ 4792us) server_negotiation.cc:407] Connection header received
0812 14:30:22.138835 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:22.138836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:22.138859 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:22.138889 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:22.140941 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.141651 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.144830 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.145421 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.145499 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:22.145510 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:22.145691 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 14:30:22.145735 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 14:30:22.146236 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:22.133575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39451 (local address 127.23.173.194:53961)
0812 14:30:22.138677 (+ 5102us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:22.138689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:22.138762 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:22.140777 (+ 2015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:22.140778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:22.140780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:22.140816 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:22.140817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.141753 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.141754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:22.144723 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:22.144726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.145545 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.145546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:22.145605 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:22.145609 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:30:22.146118 (+ 509us) client_negotiation.cc:240] Negotiation successful
0812 14:30:22.146165 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5090,"thread_start_us":68,"threads_started":1}
I20260812 14:30:22.146703 13573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20f82abaa64a20bc3bf4822c6e82c8" candidate_uuid: "a6b5119361bb400d8b75cabc43b6c121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" is_pre_election: true
I20260812 14:30:22.146862 13573 raft_consensus.cc:2454] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b5119361bb400d8b75cabc43b6c121 in term 0.
I20260812 14:30:22.146999 13570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20f82abaa64a20bc3bf4822c6e82c8" candidate_uuid: "a6b5119361bb400d8b75cabc43b6c121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74"
I20260812 14:30:22.147071 13570 raft_consensus.cc:3037] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:22.147804 13570 raft_consensus.cc:2454] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b5119361bb400d8b75cabc43b6c121 in term 1.
I20260812 14:30:22.203871 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:22.200439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.23.173.194:33229)
0812 14:30:22.200519 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:22.200521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:22.200529 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:30:22.200573 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:22.200574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:22.200596 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:22.200663 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:30:22.200848 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.201620 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.202726 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:22.203035 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:22.203106 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:22.203336 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:30:22.203477 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:30:22.203564 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:22.203761 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 14:30:22.203787 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 14:30:22.206032 13570 raft_consensus.cc:1261] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 1 FOLLOWER]: Refusing update from remote peer a6b5119361bb400d8b75cabc43b6c121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:22.206502 14174 consensus_queue.cc:1035] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3bbe4bb8224743b95a06a6cbaeae74" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:22.208504 14012 raft_consensus.cc:1261] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 1 FOLLOWER]: Refusing update from remote peer a6b5119361bb400d8b75cabc43b6c121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:22.208871 14174 consensus_queue.cc:1035] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46a2c352a4764a10add81c8a4c54d91e" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:22.217072 14191 mvcc.cc:205] Tried to move back new op lower bound from 7317688410952658944 to 7317688410695786496. Current Snapshot: MvccSnapshot[applied={T|T < 7317688410952658944}]
I20260812 14:30:22.218629 14193 mvcc.cc:205] Tried to move back new op lower bound from 7317688410952658944 to 7317688410695786496. Current Snapshot: MvccSnapshot[applied={T|T < 7317688410952658944}]
I20260812 14:30:22.220381 14198 mvcc.cc:205] Tried to move back new op lower bound from 7317688410952658944 to 7317688410695786496. Current Snapshot: MvccSnapshot[applied={T|T < 7317688410952658944}]
I20260812 14:30:22.223259 13312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:22.223363 13312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:22.224227 13312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230b1ff4efb44d0eb0560e40ba1aea49: Sending DeleteTablet for 3 replicas of tablet ba20f82abaa64a20bc3bf4822c6e82c8
I20260812 14:30:22.224720 13974 tablet_service.cc:1514] Processing DeleteTablet for tablet ba20f82abaa64a20bc3bf4822c6e82c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:22 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:50608
I20260812 14:30:22.224957 13538 tablet_service.cc:1514] Processing DeleteTablet for tablet ba20f82abaa64a20bc3bf4822c6e82c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:22 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:54304
I20260812 14:30:22.225171 13751 tablet_service.cc:1514] Processing DeleteTablet for tablet ba20f82abaa64a20bc3bf4822c6e82c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:22 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:37668
I20260812 14:30:22.225512 14208 tablet_replica.cc:330] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74: stopping tablet replica
I20260812 14:30:22.225656 14208 raft_consensus.cc:2229] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:30:22.225755 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 13379
I20260812 14:30:22.225818 14208 raft_consensus.cc:2258] T ba20f82abaa64a20bc3bf4822c6e82c8 P 2f3bbe4bb8224743b95a06a6cbaeae74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:22.235826 14206 tablet_replica.cc:330] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: stopping tablet replica
I20260812 14:30:22.235980 14206 raft_consensus.cc:2229] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:30:22.236093 14206 raft_consensus.cc:2258] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:22.260368 14206 ts_tablet_manager.cc:1905] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:22.262931 14206 ts_tablet_manager.cc:1918] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:22.263166 14206 log.cc:1196] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/ba20f82abaa64a20bc3bf4822c6e82c8
I20260812 14:30:22.263671 14206 ts_tablet_manager.cc:1939] T ba20f82abaa64a20bc3bf4822c6e82c8 P 46a2c352a4764a10add81c8a4c54d91e: Deleting consensus metadata
I20260812 14:30:22.225380 14210 tablet_replica.cc:330] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: stopping tablet replica
I20260812 14:30:22.264490 14210 raft_consensus.cc:2229] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:30:22.264743 14210 raft_consensus.cc:2258] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:22.267061 14210 ts_tablet_manager.cc:1905] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:22.269340 14210 ts_tablet_manager.cc:1918] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:22.269505 14210 log.cc:1196] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/ba20f82abaa64a20bc3bf4822c6e82c8
I20260812 14:30:22.269949 14210 ts_tablet_manager.cc:1939] T ba20f82abaa64a20bc3bf4822c6e82c8 P a6b5119361bb400d8b75cabc43b6c121: Deleting consensus metadata
I20260812 14:30:22.271019 13284 catalog_manager.cc:4841] TS a6b5119361bb400d8b75cabc43b6c121 (127.23.173.194:33229): tablet ba20f82abaa64a20bc3bf4822c6e82c8 (table test-table [id=6fca00baf2f741a8bdb9423594b1a4ad]) successfully deleted
I20260812 14:30:22.271181 13285 catalog_manager.cc:4841] TS 46a2c352a4764a10add81c8a4c54d91e (127.23.173.195:33363): tablet ba20f82abaa64a20bc3bf4822c6e82c8 (table test-table [id=6fca00baf2f741a8bdb9423594b1a4ad]) successfully deleted
W20260812 14:30:22.279482 13285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:39451 (error 108)
I20260812 14:30:22.279850 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 13656
I20260812 14:30:22.326107 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:22.325940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:39451 (local address 127.0.0.1:54314)
0812 14:30:22.326003 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:22.326055 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 14:30:22.326474 13285 catalog_manager.cc:4584] TS 2f3bbe4bb8224743b95a06a6cbaeae74 (127.23.173.193:39451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba20f82abaa64a20bc3bf4822c6e82c8: Network error: Client connection negotiation failed: client connection to 127.23.173.193:39451: connect: Connection refused (error 111)
I20260812 14:30:22.331176 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 13871
I20260812 14:30:22.354373 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 13258
I20260812 14:30:22.367986 24247 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 14:30:22.713083 24247 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 14:30:22.713169 24247 mini_ranger.cc:66] Stopping Ranger...
I20260812 14:30:23.067324 24247 mini_ranger.cc:68] Stopped Ranger
2026-08-12 14:30:23.067 UTC [11648] LOG: received smart shutdown request
2026-08-12 14:30:23.072 UTC [11648] LOG: background worker "logical replication launcher" (PID 11662) exited with exit code 1
2026-08-12 14:30:23.072 UTC [11653] LOG: shutting down
2026-08-12 14:30:23.231 UTC [11648] LOG: database system is shut down
2026-08-12T14:30:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48972 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:30:23 dist-test-slave-1l3l krb5kdc[14229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:30:23 dist-test-slave-1l3l krb5kdc[14229](info): set up 2 sockets
Aug 12 14:30:23 dist-test-slave-1l3l krb5kdc[14229](info): commencing operation
krb5kdc: starting...
W20260812 14:30:24.493002 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:30:24.608448 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35627
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35627,127.23.173.253:44163,127.23.173.252:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:24.688611 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:24.688817 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:24.688850 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:24.690006 14250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:24.690048 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:24.690066 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:24.690083 14250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:24.690099 14250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:24.691601 14250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35627,127.23.173.253:44163,127.23.173.252:33525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:24.691877 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:24.692086 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:24.696743 14264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:24.696753 14262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:24.696792 14250 server_base.cc:1034] running on GCE node
W20260812 14:30:24.696756 14261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:24.697136 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:24.697420 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:24.698565 14250 hybrid_clock.cc:648] HybridClock initialized: now 1786545024698552 us; error 28 us; skew 500 ppm
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:24.706270 14250 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:24.706744 14250 webserver.cc:460] Webserver started at http://127.23.173.254:34809/ using document root <none> and password file <none>
I20260812 14:30:24.706965 14250 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:24.707018 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:24.707146 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:24.708220 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "0fd7db6e6b844c2aa1ba4420b638c990"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "1977acfa7796f59fac7d1be984b0a53e"
server_key_iv: "d90513635b8a6eedf3a84da14086bc7c"
server_key_version: "encryptionkey@0"
I20260812 14:30:24.708549 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "0fd7db6e6b844c2aa1ba4420b638c990"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "1977acfa7796f59fac7d1be984b0a53e"
server_key_iv: "d90513635b8a6eedf3a84da14086bc7c"
server_key_version: "encryptionkey@0"
I20260812 14:30:24.709748 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:30:24.710418 14274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:24.710588 14250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:24.710671 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "0fd7db6e6b844c2aa1ba4420b638c990"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "1977acfa7796f59fac7d1be984b0a53e"
server_key_iv: "d90513635b8a6eedf3a84da14086bc7c"
server_key_version: "encryptionkey@0"
I20260812 14:30:24.710731 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:24.722297 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:24.724162 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:24.724295 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:24.728240 14250 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:35627
I20260812 14:30:24.728276 14371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:35627 every 8 connection(s)
I20260812 14:30:24.728590 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:24.729435 14372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:24.730903 14372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:30:24.732928 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 14250
I20260812 14:30:24.733036 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:30:24.733206 24247 external_mini_cluster.cc:1428] Setting key 335d86d05dbcdfb5865731c3ae9a8f14
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:24.739966 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.731548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.23.173.254:35627)
0812 14:30:24.731693 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:24.731720 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:24.731807 (+ 87us) server_negotiation.cc:407] Connection header received
0812 14:30:24.732337 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:24.732370 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:24.732388 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:24.732464 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:24.732999 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.733883 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.734699 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.735016 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.735110 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.737044 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:24.737051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:24.737051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:24.737058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:24.738489 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.738929 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.738931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.738932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.739014 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.739239 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.739239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.739239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.739415 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:24.739468 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:24.739667 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:30:24.739818 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260812 14:30:24.740043 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.731490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35627 (local address 127.0.0.1:57766)
0812 14:30:24.731647 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.731693 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:24.732191 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:24.732480 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:24.732482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:24.732728 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:24.732908 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.732913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.733972 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.733973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.734621 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.734623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.735089 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.735090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.735156 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.735607 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:24.735615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:24.736965 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:24.738579 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.738588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.738590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.738849 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.739072 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.739073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.739073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.739180 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.739471 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:24.739477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:24.739550 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 14:30:24.739650 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 14:30:24.739898 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":75,"thread_start_us":74,"threads_started":1}
I20260812 14:30:24.740796 14372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:30:24.741135 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.741024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41440)
0812 14:30:24.741073 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.741116 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:30:24.741261 14277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.173.253@KRBTEST.COM; defaulting to no policy
I20260812 14:30:24.741667 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.741572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41454)
0812 14:30:24.741617 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.741645 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:30:24.741766 14372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:44163: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab.
Entry for principal kudu/127.23.173.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab.
I20260812 14:30:24.756652 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:44163
--webserver_interface=127.23.173.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab
--principal=kudu/127.23.173.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35627,127.23.173.253:44163,127.23.173.252:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:24.790983 14372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } attempt: 1
I20260812 14:30:24.791476 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.791329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41466)
0812 14:30:24.791412 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.791446 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 14:30:24.791893 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.791805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41472)
0812 14:30:24.791844 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.791874 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:30:24.792001 14372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:44163: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
W20260812 14:30:24.836786 14382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:24.836998 14382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:24.837028 14382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:24.838140 14382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:24.838179 14382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:24.838198 14382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:24.838214 14382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:24.838229 14382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:24.839684 14382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35627,127.23.173.253:44163,127.23.173.252:33525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab
--principal=kudu/127.23.173.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:24.839948 14382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:24.840152 14382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:30:24.844848 14382 server_base.cc:1034] running on GCE node
W20260812 14:30:24.844802 14395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:24.844797 14391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:24.844797 14392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:24.845320 14382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:24.845616 14382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:24.846764 14382 hybrid_clock.cc:648] HybridClock initialized: now 1786545024846750 us; error 29 us; skew 500 ppm
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:24.849135 14382 init.cc:377] Logged in from keytab as kudu/127.23.173.253@KRBTEST.COM (short username kudu)
I20260812 14:30:24.849574 14382 webserver.cc:460] Webserver started at http://127.23.173.253:43633/ using document root <none> and password file <none>
I20260812 14:30:24.849767 14382 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:24.849814 14382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:24.849941 14382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:24.850951 14382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data/instance:
uuid: "27a06f39030a49559fab80a0b76c5689"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "875cf7fabf1628faf40aafbddb7d99a4"
server_key_iv: "64422cdaf59bac0c1de7d5bbe55132da"
server_key_version: "encryptionkey@0"
I20260812 14:30:24.851287 14382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal/instance:
uuid: "27a06f39030a49559fab80a0b76c5689"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "875cf7fabf1628faf40aafbddb7d99a4"
server_key_iv: "64422cdaf59bac0c1de7d5bbe55132da"
server_key_version: "encryptionkey@0"
I20260812 14:30:24.852641 14382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:30:24.853273 14409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:24.853415 14382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:24.853506 14382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal
uuid: "27a06f39030a49559fab80a0b76c5689"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "875cf7fabf1628faf40aafbddb7d99a4"
server_key_iv: "64422cdaf59bac0c1de7d5bbe55132da"
server_key_version: "encryptionkey@0"
I20260812 14:30:24.853569 14382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:24.860136 14372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } attempt: 2
I20260812 14:30:24.860528 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.860379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41484)
0812 14:30:24.860460 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.860498 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 14:30:24.860859 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.860770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41488)
0812 14:30:24.860813 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.860838 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:30:24.860939 14372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:44163: Network error: Client connection negotiation failed: client connection to 127.23.173.253:44163: connect: Connection refused (error 111)
I20260812 14:30:24.866907 14382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:24.868700 14382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:24.868831 14382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:24.872783 14382 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.253:44163
I20260812 14:30:24.872826 14490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.253:44163 every 8 connection(s)
I20260812 14:30:24.873116 14382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
I20260812 14:30:24.873908 14494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:24.875396 14494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:24.881383 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 14382
I20260812 14:30:24.881475 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-1/wal/instance
I20260812 14:30:24.881644 24247 external_mini_cluster.cc:1428] Setting key ad76ddd0953c02d0de208597f157b38e
I20260812 14:30:24.883970 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.876069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.23.173.254:35627)
0812 14:30:24.876127 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:24.876129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:24.876315 (+ 186us) server_negotiation.cc:407] Connection header received
0812 14:30:24.876785 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:24.876786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:24.876802 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:24.876867 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:24.877397 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.877883 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.878882 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.879090 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.879144 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.881186 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:24.881188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:24.881188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:24.881195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:24.882567 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.883019 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.883021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.883021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.883075 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.883341 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.883342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.883342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.883485 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:24.883556 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:24.883797 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 14:30:24.883883 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:30:24.884055 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.876049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35627 (local address 127.0.0.1:57772)
0812 14:30:24.876206 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.876246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:24.876670 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:24.876918 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:24.876921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:24.877148 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:24.877329 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.877333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.877981 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.877982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.878786 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.878788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.879168 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.879169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.879272 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.879782 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:24.879792 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:24.881084 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:24.882650 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.882659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.882661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.882936 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.883132 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.883133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.883133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.883265 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.883550 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:24.883555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:24.883663 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 14:30:24.883774 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 14:30:24.883894 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260812 14:30:24.884608 14494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.252@KRBTEST.COM; defaulting to no policy
I20260812 14:30:24.891891 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.884823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41502)
0812 14:30:24.884884 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.884896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:24.884937 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:24.885139 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:24.885139 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:24.885267 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:24.885329 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.885331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.886200 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.886201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.886961 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.886964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.887403 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.887404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.887476 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.887858 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:24.887863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:24.889054 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:24.890589 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.890590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.890591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.890737 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.891101 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.891102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.891102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.891173 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.891560 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:24.891561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:24.891610 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 14:30:24.891677 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:30:24.891801 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:30:24.891991 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.884862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.23.173.253:44163)
0812 14:30:24.885016 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:24.885020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:24.885038 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:30:24.885056 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:24.885058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:24.885068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:24.885126 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:24.885395 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.886111 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.887046 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.887330 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.887384 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.889131 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:24.889139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:24.889139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:24.889145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:24.890513 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.890803 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.890806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.890807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.890856 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.891231 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.891232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.891232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.891376 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:24.891588 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:24.891727 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:30:24.891879 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260812 14:30:24.892536 14494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.252@KRBTEST.COM" created.
I20260812 14:30:24.892851 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.892745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56604)
0812 14:30:24.892778 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.892823 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260812 14:30:24.892982 14413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
I20260812 14:30:24.893419 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.893315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56608)
0812 14:30:24.893359 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.893389 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:30:24.893523 14494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:33525: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab.
Entry for principal kudu/127.23.173.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab.
I20260812 14:30:24.906139 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:33525
--webserver_interface=127.23.173.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab
--principal=kudu/127.23.173.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35627,127.23.173.253:44163,127.23.173.252:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:24.942742 14494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } attempt: 1
I20260812 14:30:24.943249 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.943123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56620)
0812 14:30:24.943179 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.943220 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:30:24.943632 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.943539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56632)
0812 14:30:24.943584 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.943610 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:30:24.943754 14494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:33525: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
I20260812 14:30:24.952064 14372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } attempt: 3
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:30:24.958707 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.952352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41512)
0812 14:30:24.952407 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.952418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:24.952465 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:24.952612 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:24.952613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:24.952766 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:24.952829 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.952831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.953470 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.953471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.953972 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:24.953974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.954323 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.954324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:24.954393 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.954876 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:24.954881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:24.956338 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:24.957799 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.957800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.957800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.957957 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.958132 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:24.958133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:24.958133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:24.958188 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:24.958397 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:24.958398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:24.958446 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 14:30:24.958506 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:30:24.958613 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:30:24.958706 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.952415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.23.173.253:44163)
0812 14:30:24.952470 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:24.952472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:24.952499 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:30:24.952548 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:24.952549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:24.952559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:24.952600 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:24.952903 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.953397 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.954059 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:24.954266 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:24.954318 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:24.956418 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:24.956420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:24.956420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:24.956428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:24.957720 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.958024 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.958025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.958026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.958077 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:24.958237 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:24.958238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:24.958238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:24.958343 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:24.958397 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:24.958533 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:30:24.958613 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:30:24.959254 14372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:30:24.959571 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.959460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56640)
0812 14:30:24.959498 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.959544 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:30:24.959877 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.959799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56656)
0812 14:30:24.959835 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.959856 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:30:24.959959 14372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:33525: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
W20260812 14:30:24.988541 14518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:24.988739 14518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:24.988770 14518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:24.989878 14518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:24.989918 14518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:24.989936 14518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:24.989952 14518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:24.989967 14518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:24.991094 14372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } attempt: 1
I20260812 14:30:24.991426 14518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35627,127.23.173.253:44163,127.23.173.252:33525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab
--principal=kudu/127.23.173.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:24.991477 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.991369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56670)
0812 14:30:24.991420 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.991449 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:30:24.991669 14518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:24.991827 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:24.991733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56674)
0812 14:30:24.991781 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:24.991805 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:30:24.991897 14518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:24.991911 14372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:33525: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
W20260812 14:30:24.996388 14531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:24.996397 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:24.996513 14518 server_base.cc:1034] running on GCE node
W20260812 14:30:24.996609 14525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:24.996946 14518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:24.997222 14518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:24.998401 14518 hybrid_clock.cc:648] HybridClock initialized: now 1786545024998400 us; error 42 us; skew 500 ppm
Aug 12 14:30:24 dist-test-slave-1l3l krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:25.000728 14518 init.cc:377] Logged in from keytab as kudu/127.23.173.252@KRBTEST.COM (short username kudu)
I20260812 14:30:25.001176 14518 webserver.cc:460] Webserver started at http://127.23.173.252:41947/ using document root <none> and password file <none>
I20260812 14:30:25.001389 14518 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:25.001438 14518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:25.001567 14518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:25.002833 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data/instance:
uuid: "467dd5c3b7664c849827f8afa24f8d73"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "676ddd100f6b152a9a6b955c0e7e8736"
server_key_iv: "ba8b2d6c5eddf0fb934e66cb00dddf1c"
server_key_version: "encryptionkey@0"
I20260812 14:30:25.003175 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal/instance:
uuid: "467dd5c3b7664c849827f8afa24f8d73"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "676ddd100f6b152a9a6b955c0e7e8736"
server_key_iv: "ba8b2d6c5eddf0fb934e66cb00dddf1c"
server_key_version: "encryptionkey@0"
I20260812 14:30:25.004470 14518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:30:25.005151 14541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:25.005298 14518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:25.005374 14518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal
uuid: "467dd5c3b7664c849827f8afa24f8d73"
format_stamp: "Formatted at 2026-08-12 14:30:24 on dist-test-slave-1l3l"
server_key: "676ddd100f6b152a9a6b955c0e7e8736"
server_key_iv: "ba8b2d6c5eddf0fb934e66cb00dddf1c"
server_key_version: "encryptionkey@0"
I20260812 14:30:25.005438 14518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:25.011888 14494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } attempt: 2
I20260812 14:30:25.012291 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.012166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56686)
0812 14:30:25.012214 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.012256 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:30:25.012646 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.012571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56692)
0812 14:30:25.012606 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.012629 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:30:25.012732 14494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:33525: Network error: Client connection negotiation failed: client connection to 127.23.173.252:33525: connect: Connection refused (error 111)
I20260812 14:30:25.022019 14518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:25.023684 14518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:25.023842 14518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:25.028854 14518 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.252:33525
I20260812 14:30:25.028898 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.252:33525 every 8 connection(s)
I20260812 14:30:25.029279 14518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
I20260812 14:30:25.030381 14626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:25.030457 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 14518
I20260812 14:30:25.030565 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544840008271-24247-0/minicluster-data/master-2/wal/instance
I20260812 14:30:25.030754 24247 external_mini_cluster.cc:1428] Setting key 4d47f73a25413f00b041bf762454ad1c
I20260812 14:30:25.032284 14626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:30:25.042009 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.033237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.23.173.254:35627)
0812 14:30:25.033281 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.033283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.033496 (+ 213us) server_negotiation.cc:407] Connection header received
0812 14:30:25.033560 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.033561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.033572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.033645 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.034071 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.034778 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.035963 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.036233 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.036293 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.038535 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.038537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.038537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.038545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.040862 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.041233 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.041234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.041234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.041282 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.041487 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.041488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.041489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.041598 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.041654 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.041833 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.041918 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:30:25.042189 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.031411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.23.173.252:33525)
0812 14:30:25.031899 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.031910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.031969 (+ 59us) server_negotiation.cc:407] Connection header received
0812 14:30:25.032538 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.032547 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.032611 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.032753 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.033509 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.034687 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.035775 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.036163 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.036264 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.039015 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.039024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.039025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.039032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.040694 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.041067 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.041070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.041070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.041149 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.041359 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.041360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.041360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.041600 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.041738 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.041833 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.041978 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":388,"thread_start_us":80,"threads_started":1}
I20260812 14:30:25.042253 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.033199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:35627 (local address 127.0.0.1:57780)
0812 14:30:25.033413 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.033448 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:25.033509 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:25.033662 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:25.033664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:25.033899 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:25.033990 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.033994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.034868 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.034869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.035896 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.035899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.036306 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.036306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.036374 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.036833 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:25.036843 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:25.038449 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:25.040936 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.040944 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.040947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.041164 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.041339 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.041340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.041340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.041430 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.041651 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:25.041659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:25.041727 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 14:30:25.041805 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:30:25.042131 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":160,"thread_start_us":88,"threads_started":1}
I20260812 14:30:25.042781 14626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:25.049875 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.043101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.23.173.253:44163)
0812 14:30:25.043164 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.043166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.043174 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:30:25.043196 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.043197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.043206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.043253 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.043503 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.044170 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.045048 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.045241 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.045301 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.047102 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.047103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.047104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.047111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.048826 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.049183 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.049184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.049184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.049231 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.049416 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.049417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.049418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.049527 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.049586 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.049717 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.049787 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:30:25.049891 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.042997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:44163 (local address 127.0.0.1:41518)
0812 14:30:25.043035 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.043045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:25.043083 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:25.043272 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:25.043272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:25.043399 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:25.043436 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.043438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.044255 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.044256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.044976 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.044979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.045304 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.045304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.045345 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.045683 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:25.045687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:25.047016 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:25.048918 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.048919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.048919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.049111 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.049283 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.049284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.049284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.049347 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.049585 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:25.049586 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:25.049627 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 14:30:25.049687 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:30:25.049774 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260812 14:30:25.050377 14626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:30:25.050436 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.043050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.23.173.254:35627)
0812 14:30:25.043089 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.043091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.043098 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:25.043116 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.043117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.043125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.043178 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.043727 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.044347 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.045221 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.045461 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.045516 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.047799 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.047800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.047800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.047806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.049236 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.049573 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.049574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.049575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.049622 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.049785 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.049786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.049786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.049880 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.050067 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.050271 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.050354 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:30:25.056553 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.050628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56696)
0812 14:30:25.050674 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.050685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:25.050724 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:25.050847 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:25.050848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:25.050979 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:25.051009 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.051011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.051577 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.051577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.052082 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.052083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.052399 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.052400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.052453 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.052772 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:25.052776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:25.053873 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:25.055406 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.055407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.055407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.055566 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.055916 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.055917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.055917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.055969 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.056250 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:25.056251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:25.056297 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:30:25.056367 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:30:25.056467 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:30:25.056636 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.050721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.23.173.252:33525)
0812 14:30:25.050760 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.050762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.050769 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:25.050784 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.050784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.050791 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.050834 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.051075 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.051515 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.052154 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.052348 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.052396 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.053963 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.053964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.053965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.053970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.055312 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.055634 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.055635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.055635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.055689 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.056048 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.056049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.056050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.056167 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.056229 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.056399 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.056546 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":22}
I20260812 14:30:25.057945 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.051011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.23.173.253:44163)
0812 14:30:25.051052 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.051054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.051061 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:25.051079 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.051079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.051087 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.051130 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.051537 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.052012 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.052623 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.052811 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.052862 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.055466 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.055467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.055468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.055474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.056938 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.057243 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.057244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.057245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.057303 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.057462 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.057463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.057463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.057580 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.057753 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.057793 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.057866 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:30:25.058122 14626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73: Bootstrap starting.
I20260812 14:30:25.059078 14626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:25.059279 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:30:25.059368 14626 log.cc:824] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:25.060151 14626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73: No bootstrap required, opened a new log
I20260812 14:30:25.061957 14626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.062105 14626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:25.062135 14626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467dd5c3b7664c849827f8afa24f8d73, State: Initialized, Role: FOLLOWER
I20260812 14:30:25.062243 14626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.062502 14643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.062549 14626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:25.064038 14643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:30:25.065586 14668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:30:25.065644 14668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:30:25.067044 14372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } attempt: 2
I20260812 14:30:25.064764 14518 master_runner.cc:186] Error getting master registration for 127.23.173.254:35627: OK, instance_id { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" instance_seqno: 1786545024727144 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:30:25.067652 14518 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:30:25.068646 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.060609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.23.173.254:35627)
0812 14:30:25.060681 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.060682 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.060936 (+ 254us) server_negotiation.cc:407] Connection header received
0812 14:30:25.061101 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.061102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.061111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.061161 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.061664 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.062112 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.063049 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.063313 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.063358 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.064825 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.064827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.064827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.064833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.066478 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.067542 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.067543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.067544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.067598 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.067813 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.067814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.067814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.068174 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.068262 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.068487 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.068563 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 14:30:25.069730 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.060607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.23.173.252:33525)
0812 14:30:25.060646 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.060647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.060875 (+ 228us) server_negotiation.cc:407] Connection header received
0812 14:30:25.060976 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.060977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.060988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.061047 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.061603 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.062228 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.063161 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.063452 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.063490 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.065086 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.065087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.065088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.065094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.067152 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.068659 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.068660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.068660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.068938 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.069157 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.069158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.069158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.069301 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.069365 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.069578 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.069649 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:30:25.070860 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.060686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.23.173.253:44163)
0812 14:30:25.060718 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.060719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.062698 (+ 1979us) server_negotiation.cc:407] Connection header received
0812 14:30:25.062784 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.062784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.062793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.062818 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.063486 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.064660 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.066158 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.066351 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.066412 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.067726 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.067727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.067727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.067734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.069482 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.070042 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.070044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.070044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.070103 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.070332 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.070333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.070334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.070467 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.070557 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.070697 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.070773 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:30:25.075174 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.067860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.23.173.252:33525)
0812 14:30:25.068021 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.068022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.068030 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:30:25.068048 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.068049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.068058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.068122 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.069070 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.069798 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.070639 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.070930 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.071213 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.072905 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.072906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.072906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.072913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.074182 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.074566 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.074566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.074566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.074609 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.074778 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.074779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.074779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.074862 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.074917 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.075054 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.075108 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260812 14:30:25.075224 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.067260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56712)
0812 14:30:25.067299 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.067310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:25.067349 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:25.068131 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:25.068132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:25.068277 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:25.068315 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.068316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.069891 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.069903 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.070545 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.070548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.071020 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.071022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.071077 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.071440 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:25.071445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:25.072811 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:25.074268 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.074270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.074271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.074494 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.074663 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.074663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.074664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.074724 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.074921 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:25.074922 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:25.074965 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:30:25.075039 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:30:25.075130 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:30:25.076650 14372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: Bootstrap starting.
I20260812 14:30:25.077374 14372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:25.077639 14372 log.cc:824] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:25.078454 14372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: No bootstrap required, opened a new log
I20260812 14:30:25.080485 14372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.080631 14372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:25.080673 14372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd7db6e6b844c2aa1ba4420b638c990, State: Initialized, Role: FOLLOWER
I20260812 14:30:25.080790 14372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.081063 14674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.081116 14372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:25.081142 14674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:30:25.083312 14690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:30:25.083384 14690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:30:25.083791 14250 master_runner.cc:186] Error getting master registration for 127.23.173.253:44163: OK, instance_id { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" instance_seqno: 1786545024871682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:30:25.086601 14250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:30:25.103879 14494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } attempt: 3
Aug 12 14:30:25 dist-test-slave-1l3l krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:30:25.111927 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.104151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:33525 (local address 127.0.0.1:56716)
0812 14:30:25.104200 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:25.104211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:25.104245 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:25.104374 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:25.104375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:25.104553 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:25.104600 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.104602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.105519 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.105520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.106326 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:25.106329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.106806 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.106807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:25.106866 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.107347 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:25.107353 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:25.108927 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:25.110733 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.110734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.110735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.110982 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.111255 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:25.111256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:25.111256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:25.111315 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:25.111584 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:25.111585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:25.111630 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:30:25.111688 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:30:25.111815 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:30:25.112388 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:25.104199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.23.173.252:33525)
0812 14:30:25.104238 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:25.104240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:25.104249 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:25.104296 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:25.104297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:25.104306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:25.104365 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:25.104664 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.105428 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.106407 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:25.106728 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:25.106798 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:25.109002 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:25.109004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:25.109004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:25.109012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:25.110638 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.111127 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.111128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.111129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.111189 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:25.111375 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:25.111375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:25.111376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:25.111501 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:25.111559 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:25.112095 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 14:30:25.112177 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:30:25.113902 14494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: Bootstrap starting.
I20260812 14:30:25.114636 14494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:25.114939 14494 log.cc:824] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:25.116313 14494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: No bootstrap required, opened a new log
I20260812 14:30:25.118287 14494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.118418 14494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:25.118450 14494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a06f39030a49559fab80a0b76c5689, State: Initialized, Role: FOLLOWER
I20260812 14:30:25.118556 14494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.118839 14494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 14:30:25.121124 14717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:30:25.121172 14717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:30:25.121338 14695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.121400 14695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:30:25.123823 14417 tablet.cc:2369] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:30:25.126638 14382 master_runner.cc:263] No leader master found from master 27a06f39030a49559fab80a0b76c5689
I20260812 14:30:25.126686 14382 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:30:25.138230 14643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:25.138330 14643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.138770 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467dd5c3b7664c849827f8afa24f8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a06f39030a49559fab80a0b76c5689" is_pre_election: true
I20260812 14:30:25.138929 14440 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467dd5c3b7664c849827f8afa24f8d73 in term 0.
I20260812 14:30:25.139106 14542 leader_election.cc:302] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a06f39030a49559fab80a0b76c5689, 467dd5c3b7664c849827f8afa24f8d73; no voters:
I20260812 14:30:25.139571 14318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467dd5c3b7664c849827f8afa24f8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" is_pre_election: true
I20260812 14:30:25.139748 14318 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467dd5c3b7664c849827f8afa24f8d73 in term 0.
I20260812 14:30:25.140143 14643 leader_election.cc:288] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fd7db6e6b844c2aa1ba4420b638c990 (127.23.173.254:35627), 27a06f39030a49559fab80a0b76c5689 (127.23.173.253:44163)
I20260812 14:30:25.140237 14643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:30:25.140273 14643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:30:25.140292 14643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:25.141243 14643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.141482 14318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467dd5c3b7664c849827f8afa24f8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd7db6e6b844c2aa1ba4420b638c990"
I20260812 14:30:25.141557 14318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:25.142496 14318 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467dd5c3b7664c849827f8afa24f8d73 in term 1.
I20260812 14:30:25.142728 14548 leader_election.cc:302] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fd7db6e6b844c2aa1ba4420b638c990, 467dd5c3b7664c849827f8afa24f8d73; no voters:
I20260812 14:30:25.143110 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467dd5c3b7664c849827f8afa24f8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a06f39030a49559fab80a0b76c5689"
I20260812 14:30:25.143193 14440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:25.144127 14440 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467dd5c3b7664c849827f8afa24f8d73 in term 1.
I20260812 14:30:25.144541 14643 leader_election.cc:288] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [CANDIDATE]: Term 1 election: Requested vote from peers 0fd7db6e6b844c2aa1ba4420b638c990 (127.23.173.254:35627), 27a06f39030a49559fab80a0b76c5689 (127.23.173.253:44163)
I20260812 14:30:25.144822 14643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:25.145000 14643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [term 1 LEADER]: Becoming Leader. State: Replica: 467dd5c3b7664c849827f8afa24f8d73, State: Running, Role: LEADER
I20260812 14:30:25.145252 14643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } }
I20260812 14:30:25.145928 14643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467dd5c3b7664c849827f8afa24f8d73. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.146096 14643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:25.146888 14722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:25.147167 14722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:25.148878 14440 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [term 1 FOLLOWER]: Refusing update from remote peer 467dd5c3b7664c849827f8afa24f8d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:25.149290 14720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:25.150794 14695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467dd5c3b7664c849827f8afa24f8d73. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.150883 14695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:30:25.152318 14318 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [term 1 FOLLOWER]: Refusing update from remote peer 467dd5c3b7664c849827f8afa24f8d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:25.152626 14720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:25.153823 14674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467dd5c3b7664c849827f8afa24f8d73. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.153921 14674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:30:25.155234 14720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.155314 14720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:25.155943 14695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.156036 14695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:30:25.156304 14723 mvcc.cc:205] Tried to move back new op lower bound from 7317688423007285248 to 7317688422996594688. Current Snapshot: MvccSnapshot[applied={T|T < 7317688423007285248}]
I20260812 14:30:25.156522 14725 mvcc.cc:205] Tried to move back new op lower bound from 7317688423007285248 to 7317688422996594688. Current Snapshot: MvccSnapshot[applied={T|T < 7317688423007285248}]
I20260812 14:30:25.157691 14643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.157769 14643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:25.158254 14674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "467dd5c3b7664c849827f8afa24f8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd7db6e6b844c2aa1ba4420b638c990" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35627 } } peers { permanent_uuid: "27a06f39030a49559fab80a0b76c5689" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 44163 } } peers { permanent_uuid: "467dd5c3b7664c849827f8afa24f8d73" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 33525 } } }
I20260812 14:30:25.158335 14674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:30:25.158596 14728 mvcc.cc:205] Tried to move back new op lower bound from 7317688423007285248 to 7317688422996594688. Current Snapshot: MvccSnapshot[applied={T|T < 7317688423007285248}]
I20260812 14:30:25.159587 14722 catalog_manager.cc:1347] Generated new cluster ID: 36c778281ca048f89d92eb7017dee38e
I20260812 14:30:25.159636 14722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:30:25.183720 14722 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:30:25.184512 14722 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:25.199316 14722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467dd5c3b7664c849827f8afa24f8d73: Generated new TSK 0
I20260812 14:30:25.199581 14722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:30:26.083889 14690 catalog_manager.cc:1259] Loaded cluster ID: 36c778281ca048f89d92eb7017dee38e
I20260812 14:30:26.083982 14690 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: loading cluster ID for follower catalog manager: success
I20260812 14:30:26.084830 14690 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: acquiring CA information for follower catalog manager: success
I20260812 14:30:26.085140 14690 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fd7db6e6b844c2aa1ba4420b638c990: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:30:26.121692 14717 catalog_manager.cc:1259] Loaded cluster ID: 36c778281ca048f89d92eb7017dee38e
I20260812 14:30:26.121778 14717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: loading cluster ID for follower catalog manager: success
I20260812 14:30:26.122721 14717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: acquiring CA information for follower catalog manager: success
I20260812 14:30:26.123085 14717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27a06f39030a49559fab80a0b76c5689: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:30:26.262918 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 14250
I20260812 14:30:26.290132 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 14382
I20260812 14:30:26.305055 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 14518
2026-08-12T14:30:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2982 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:30:26 dist-test-slave-1l3l krb5kdc[14742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:30:26 dist-test-slave-1l3l krb5kdc[14742](info): set up 2 sockets
Aug 12 14:30:26 dist-test-slave-1l3l krb5kdc[14742](info): commencing operation
krb5kdc: starting...
W20260812 14:30:27.471098 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:30:27 dist-test-slave-1l3l krb5kdc[14742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:30:27.584560 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46477
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:27.665557 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:27.665750 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:27.665769 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:27.666858 14759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:27.666899 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:27.666913 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:27.666935 14759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:27.666957 14759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:27.668428 14759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:27.668674 14759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:27.668881 14759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:27.673439 14767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:27.673465 14768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:27.673477 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:27.673635 14759 server_base.cc:1034] running on GCE node
I20260812 14:30:27.673856 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:27.674125 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:27.675261 14759 hybrid_clock.cc:648] HybridClock initialized: now 1786545027675247 us; error 28 us; skew 500 ppm
Aug 12 14:30:27 dist-test-slave-1l3l krb5kdc[14742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545027, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:27.677610 14759 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:27.677984 14759 webserver.cc:460] Webserver started at http://127.23.173.254:37269/ using document root <none> and password file <none>
I20260812 14:30:27.678155 14759 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:27.678195 14759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:27.678287 14759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:27.679260 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "52de5857a3754129a83b2d935e5b41c4"
format_stamp: "Formatted at 2026-08-12 14:30:27 on dist-test-slave-1l3l"
server_key: "3619ee80001b0ac43d5ae572b8fefacc"
server_key_iv: "d0852dfaf9a38cf2c46328491cb718e2"
server_key_version: "encryptionkey@0"
I20260812 14:30:27.679626 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "52de5857a3754129a83b2d935e5b41c4"
format_stamp: "Formatted at 2026-08-12 14:30:27 on dist-test-slave-1l3l"
server_key: "3619ee80001b0ac43d5ae572b8fefacc"
server_key_iv: "d0852dfaf9a38cf2c46328491cb718e2"
server_key_version: "encryptionkey@0"
I20260812 14:30:27.680866 14759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:30:27.681489 14786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:27.681650 14759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:27.681726 14759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "52de5857a3754129a83b2d935e5b41c4"
format_stamp: "Formatted at 2026-08-12 14:30:27 on dist-test-slave-1l3l"
server_key: "3619ee80001b0ac43d5ae572b8fefacc"
server_key_iv: "d0852dfaf9a38cf2c46328491cb718e2"
server_key_version: "encryptionkey@0"
I20260812 14:30:27.681778 14759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:27.696758 14759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:27.698407 14759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:27.698511 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:27.702464 14759 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46477
I20260812 14:30:27.702495 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46477 every 8 connection(s)
I20260812 14:30:27.702805 14759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:27.703655 14885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:27.705873 14885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4: Bootstrap starting.
I20260812 14:30:27.706432 14885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:27.706705 14885 log.cc:824] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:27.707376 14885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4: No bootstrap required, opened a new log
I20260812 14:30:27.709033 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 14759
I20260812 14:30:27.708976 14885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52de5857a3754129a83b2d935e5b41c4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46477 } }
I20260812 14:30:27.709117 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:30:27.709110 14885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:27.709152 14885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52de5857a3754129a83b2d935e5b41c4, State: Initialized, Role: FOLLOWER
I20260812 14:30:27.709302 24247 external_mini_cluster.cc:1428] Setting key 1c33c4aa2a3120ee1770cf5892d4d0e6
I20260812 14:30:27.709290 14885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52de5857a3754129a83b2d935e5b41c4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46477 } }
I20260812 14:30:27.709371 14885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:27.709410 14885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:27.709506 14885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:27.710562 14885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52de5857a3754129a83b2d935e5b41c4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46477 } }
I20260812 14:30:27.710652 14885 leader_election.cc:302] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52de5857a3754129a83b2d935e5b41c4; no voters:
I20260812 14:30:27.710785 14885 leader_election.cc:288] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:30:27.710837 14894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:27.711030 14885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:27.711041 14894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [term 1 LEADER]: Becoming Leader. State: Replica: 52de5857a3754129a83b2d935e5b41c4, State: Running, Role: LEADER
I20260812 14:30:27.711187 14894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52de5857a3754129a83b2d935e5b41c4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46477 } }
I20260812 14:30:27.711539 14894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52de5857a3754129a83b2d935e5b41c4. Latest consensus state: current_term: 1 leader_uuid: "52de5857a3754129a83b2d935e5b41c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52de5857a3754129a83b2d935e5b41c4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46477 } } }
I20260812 14:30:27.711616 14894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:27.711791 14895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52de5857a3754129a83b2d935e5b41c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52de5857a3754129a83b2d935e5b41c4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46477 } } }
I20260812 14:30:27.711865 14895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:27.712531 14905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:27.713169 14905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:27.714814 14905 catalog_manager.cc:1347] Generated new cluster ID: ee1b9e830f6a4c6eaf7af344d8fd7480
I20260812 14:30:27.714859 14905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:30:27 dist-test-slave-1l3l krb5kdc[14742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:27.720304 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:27.709715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.23.173.254:46477)
0812 14:30:27.709906 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:27.709916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:27.709978 (+ 62us) server_negotiation.cc:407] Connection header received
0812 14:30:27.710124 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:27.710132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:27.710190 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:27.710333 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:27.711089 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:27.712457 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:27.713310 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:27.713744 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:27.713851 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:27.716696 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:27.716700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:27.716700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:27.716713 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:27.718678 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:27.719047 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:27.719050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:27.719051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:27.719169 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:27.719352 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:27.719353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:27.719353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:27.719737 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:27.719894 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:27.719966 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:30:27.720107 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260812 14:30:27.721215 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:30:27.730803 14905 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:30:27.731266 14905 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:27.734200 14905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52de5857a3754129a83b2d935e5b41c4: Generated new TSK 0
I20260812 14:30:27.734359 14905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:30:27.775043 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 14759
2026-08-12T14:30:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1469 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:30:27 dist-test-slave-1l3l krb5kdc[14926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:30:27 dist-test-slave-1l3l krb5kdc[14926](info): set up 2 sockets
Aug 12 14:30:27 dist-test-slave-1l3l krb5kdc[14926](info): commencing operation
krb5kdc: starting...
W20260812 14:30:28.940078 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:30:29.058421 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:38695
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:29.139954 14948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:29.140232 14948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:29.140265 14948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:29.141989 14948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:29.142110 14948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:29.142158 14948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:29.142194 14948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:29.142226 14948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:29.144686 14948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:29.145776 14948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:29.146023 14948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:29.151937 14956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.151969 14948 server_base.cc:1034] running on GCE node
W20260812 14:30:29.151988 14963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:29.152179 14961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.152477 14948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:29.152832 14948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:29.154011 14948 hybrid_clock.cc:648] HybridClock initialized: now 1786545029153979 us; error 45 us; skew 500 ppm
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:29.157765 14948 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:29.158493 14948 webserver.cc:460] Webserver started at http://127.23.173.254:38317/ using document root <none> and password file <none>
I20260812 14:30:29.158728 14948 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:29.158795 14948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:29.158934 14948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:29.160365 14948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "33d9896a3e2047caa7da5b80a8fc5564"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "983f4baa0bf280936b0f2ea42f09b150"
server_key_iv: "aeff868a5a591d705187e745cefc3e13"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.160790 14948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "33d9896a3e2047caa7da5b80a8fc5564"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "983f4baa0bf280936b0f2ea42f09b150"
server_key_iv: "aeff868a5a591d705187e745cefc3e13"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.162252 14948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:30:29.162997 14981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.163153 14948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:29.163250 14948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "33d9896a3e2047caa7da5b80a8fc5564"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "983f4baa0bf280936b0f2ea42f09b150"
server_key_iv: "aeff868a5a591d705187e745cefc3e13"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.163322 14948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:29.175886 14948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:29.178450 14948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:29.178601 14948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:29.183866 14948 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:38695
I20260812 14:30:29.183967 15063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:38695 every 8 connection(s)
I20260812 14:30:29.184333 14948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:29.184571 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 14948
I20260812 14:30:29.184679 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:30:29.184880 24247 external_mini_cluster.cc:1428] Setting key b215618021d8aab94125048e05239b7a
I20260812 14:30:29.185482 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:29.188635 15065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap starting.
I20260812 14:30:29.189328 15065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:29.189637 15065 log.cc:824] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:29.190292 15065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: No bootstrap required, opened a new log
I20260812 14:30:29.192400 15065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:29.192572 15065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:29.192618 15065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d9896a3e2047caa7da5b80a8fc5564, State: Initialized, Role: FOLLOWER
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:29.192744 15065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:29.192842 15065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:29.192879 15065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:29.192926 15065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:29.193563 15065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:29.193671 15065 leader_election.cc:302] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33d9896a3e2047caa7da5b80a8fc5564; no voters:
I20260812 14:30:29.193843 15065 leader_election.cc:288] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:30:29.193874 15072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:29.194109 15065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:29.194253 15072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 LEADER]: Becoming Leader. State: Replica: 33d9896a3e2047caa7da5b80a8fc5564, State: Running, Role: LEADER
I20260812 14:30:29.194350 15072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:29.194716 15073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33d9896a3e2047caa7da5b80a8fc5564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } } }
I20260812 14:30:29.194736 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d9896a3e2047caa7da5b80a8fc5564. Latest consensus state: current_term: 1 leader_uuid: "33d9896a3e2047caa7da5b80a8fc5564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } } }
I20260812 14:30:29.194799 15073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:29.194808 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:29.194989 15083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:29.195549 15083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:29.197062 15083 catalog_manager.cc:1347] Generated new cluster ID: c5d2d369d61f491588621ac152fcaac7
I20260812 14:30:29.197124 15083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:30:29.197257 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.185633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.23.173.254:38695)
0812 14:30:29.185801 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.185810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.185840 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:30:29.186317 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.186325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.186386 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.186538 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:29.187319 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.188674 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.189758 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.190345 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.190484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.193231 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:29.193234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:29.193235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:29.193245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:29.195045 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.195429 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.195432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.195433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.195544 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.195892 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.195893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.195894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.196185 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:29.196307 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.196839 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.196970 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260812 14:30:29.215551 15083 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:30:29.216079 15083 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:29.222280 15083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Generated new TSK 0
I20260812 14:30:29.222438 15083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:30:29.256465 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:38695
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:29.353308 15102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:29.353495 15102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:29.353518 15102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:29.354607 15102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:29.354646 15102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:29.354674 15102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:30:29.356118 15102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:29.356328 15102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:29.356532 15102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:30:29.361268 15102 server_base.cc:1034] running on GCE node
W20260812 14:30:29.361248 15112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:29.361312 15113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:29.361253 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.361714 15102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:29.361907 15102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:29.363034 15102 hybrid_clock.cc:648] HybridClock initialized: now 1786545029363021 us; error 31 us; skew 500 ppm
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:29.365391 15102 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:30:29.365849 15102 webserver.cc:460] Webserver started at http://127.23.173.193:40997/ using document root <none> and password file <none>
I20260812 14:30:29.366058 15102 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:29.366107 15102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:29.366240 15102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:29.367242 15102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "365a7921834e2f96bd0d610dc4d5defe"
server_key_iv: "02171e46183cd1f489b21af9d210ad19"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.367568 15102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "365a7921834e2f96bd0d610dc4d5defe"
server_key_iv: "02171e46183cd1f489b21af9d210ad19"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.368793 15102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:30:29.369424 15128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.369573 15102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:29.369652 15102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "365a7921834e2f96bd0d610dc4d5defe"
server_key_iv: "02171e46183cd1f489b21af9d210ad19"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.369714 15102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:29.382822 15102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:29.384464 15102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:29.384601 15102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:29.384799 15102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:29.385072 15102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:29.385105 15102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.385133 15102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:29.385150 15102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.390661 15102 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:38443
I20260812 14:30:29.390717 15326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:38443 every 8 connection(s)
I20260812 14:30:29.390992 15102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:30:29.391899 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 15102
I20260812 14:30:29.392020 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:30:29.392199 24247 external_mini_cluster.cc:1428] Setting key 1c70530ba96405bc97274b27eefff4d4
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:29.400885 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.391645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:43325 (local address 127.23.173.254:38695)
0812 14:30:29.391697 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.391699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.391883 (+ 184us) server_negotiation.cc:407] Connection header received
0812 14:30:29.392449 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.392451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.392464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.392519 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:29.393506 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.394116 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.395519 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.395854 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.395910 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.398269 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:29.398271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:29.398271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:29.398278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:29.399534 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.400017 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.400017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.400018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.400067 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.400316 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.400316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.400316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.400406 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:29.400450 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.400757 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.400821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:30:29.401129 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.391607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:43325)
0812 14:30:29.391776 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.391811 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.392331 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.392592 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.392596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.393227 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:29.393411 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.393416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.394207 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.394208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.395443 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.395446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.395942 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.395943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.396076 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.396526 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:29.396543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:29.398177 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:29.399613 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:29.399621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:29.399623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:29.399945 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:29.400123 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:29.400124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:29.400125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:29.400258 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:29.400458 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:29.400464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:29.400631 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.400749 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.400910 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
I20260812 14:30:29.401692 15327 heartbeater.cc:347] Connected to a master server at 127.23.173.254:38695
I20260812 14:30:29.401782 15327 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:29.401948 15327 heartbeater.cc:511] Master 127.23.173.254:38695 requested a full tablet report, sending...
I20260812 14:30:29.402444 15004 ts_manager.cc:194] Registered new tserver with Master: 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443)
I20260812 14:30:29.403055 15004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:43325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:30:29.418253 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:38695
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:29.496961 15336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:29.497162 15336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:29.497193 15336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:29.498297 15336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:29.498338 15336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:29.498373 15336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:30:29.499797 15336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:29.500017 15336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:29.500228 15336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:29.505002 15343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.505070 15336 server_base.cc:1034] running on GCE node
W20260812 14:30:29.505012 15344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:29.504976 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.505398 15336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:29.505623 15336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:29.506757 15336 hybrid_clock.cc:648] HybridClock initialized: now 1786545029506738 us; error 37 us; skew 500 ppm
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:29.509230 15336 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:30:29.509653 15336 webserver.cc:460] Webserver started at http://127.23.173.194:43055/ using document root <none> and password file <none>
I20260812 14:30:29.509850 15336 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:29.509893 15336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:29.510020 15336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:29.510982 15336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "d5ff9cbb51dd6042e11d393b25395a1c"
server_key_iv: "b99927244ac306c6488dd03655dcdf52"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.511307 15336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "d5ff9cbb51dd6042e11d393b25395a1c"
server_key_iv: "b99927244ac306c6488dd03655dcdf52"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.512588 15336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:30:29.513264 15360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.513415 15336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:29.513504 15336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "d5ff9cbb51dd6042e11d393b25395a1c"
server_key_iv: "b99927244ac306c6488dd03655dcdf52"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.513568 15336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:29.524519 15336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:29.526471 15336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:29.526602 15336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:29.526799 15336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:29.527073 15336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:29.527112 15336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.527138 15336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:29.527155 15336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.532629 15336 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:39383
I20260812 14:30:29.532682 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:39383 every 8 connection(s)
I20260812 14:30:29.532953 15336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:29.541030 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.533592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:45753 (local address 127.23.173.254:38695)
0812 14:30:29.533651 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.533653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.533807 (+ 154us) server_negotiation.cc:407] Connection header received
0812 14:30:29.534292 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.534294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.534306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.534359 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:29.535238 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.535784 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.536715 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.536904 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.536959 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.538772 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:29.538774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:29.538774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:29.538782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:29.539836 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.540194 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.540196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.540196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.540250 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.540457 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.540457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.540458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.540571 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:29.540627 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.540865 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.540942 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:30:29.541183 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.533531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:45753)
0812 14:30:29.533698 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.533736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.534194 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.534431 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.534435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.534960 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:29.535143 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.535147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.535865 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.535866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.536631 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.536633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.536973 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.536973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.537091 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.537459 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:29.537466 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:29.538684 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:29.539911 (+ 1227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:29.539918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:29.539920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:29.540127 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:29.540306 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:29.540307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:29.540307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:29.540402 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:29.540627 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:29.540634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:29.540754 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.540838 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.540980 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260812 14:30:29.541616 15568 heartbeater.cc:347] Connected to a master server at 127.23.173.254:38695
I20260812 14:30:29.541697 15568 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:29.541831 15568 heartbeater.cc:511] Master 127.23.173.254:38695 requested a full tablet report, sending...
I20260812 14:30:29.542158 15004 ts_manager.cc:194] Registered new tserver with Master: 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383)
I20260812 14:30:29.542505 15004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:45753
I20260812 14:30:29.542778 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 15336
I20260812 14:30:29.542886 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:30:29.543072 24247 external_mini_cluster.cc:1428] Setting key ffd5b6917bf74a68cb3713110f137036
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:30:29.566032 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:38695
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:29.646449 15574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:29.646657 15574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:29.646687 15574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:29.647838 15574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:29.647878 15574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:29.647913 15574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:30:29.649325 15574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:29.649552 15574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:29.649753 15574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:29.654456 15580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:29.654470 15588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.654505 15574 server_base.cc:1034] running on GCE node
W20260812 14:30:29.654500 15579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:29.654888 15574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:29.655115 15574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:29.656248 15574 hybrid_clock.cc:648] HybridClock initialized: now 1786545029656231 us; error 33 us; skew 500 ppm
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:29.658514 15574 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:30:29.658991 15574 webserver.cc:460] Webserver started at http://127.23.173.195:32927/ using document root <none> and password file <none>
I20260812 14:30:29.659198 15574 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:29.659246 15574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:29.659372 15574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:29.660400 15574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "90d9d5f039e841c394cebd17bd4b6f80"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "a65e1f3c7df8b65243a3f9b581047ac4"
server_key_iv: "403e688784545e5a6f950b5f19daca98"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.660728 15574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "90d9d5f039e841c394cebd17bd4b6f80"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "a65e1f3c7df8b65243a3f9b581047ac4"
server_key_iv: "403e688784545e5a6f950b5f19daca98"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.661933 15574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:30:29.662617 15598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.662766 15574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:29.662847 15574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "90d9d5f039e841c394cebd17bd4b6f80"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "a65e1f3c7df8b65243a3f9b581047ac4"
server_key_iv: "403e688784545e5a6f950b5f19daca98"
server_key_version: "encryptionkey@0"
I20260812 14:30:29.662909 15574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:29.678514 15574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:29.680189 15574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:29.680320 15574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:29.680523 15574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:29.680799 15574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:29.680833 15574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.680860 15574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:29.680878 15574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:29.686404 15574 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:46253
I20260812 14:30:29.686443 15788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:46253 every 8 connection(s)
I20260812 14:30:29.686720 15574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:30:29.690294 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 15574
I20260812 14:30:29.690397 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:30:29.690585 24247 external_mini_cluster.cc:1428] Setting key 8c74351657d29c786989d39fab2e50ee
Aug 12 14:30:29 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:29.695899 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.687346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:53813 (local address 127.23.173.254:38695)
0812 14:30:29.687396 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.687398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.687592 (+ 194us) server_negotiation.cc:407] Connection header received
0812 14:30:29.688132 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.688133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.688145 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.688196 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:29.689094 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.689579 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.690755 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.691019 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.691094 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.693236 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:29.693237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:29.693237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:29.693244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:29.694438 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.694846 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.694848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.694848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.694915 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.695158 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.695160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.695160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.695279 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:29.695409 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.695686 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.695801 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:30:29.696053 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.687306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:53813)
0812 14:30:29.687482 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.687518 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.688041 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.688278 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.688282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.688819 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:29.689004 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.689009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.689660 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.689662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.690659 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.690662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.691108 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.691109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.691243 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.691753 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:29.691762 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:29.693141 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:29.694510 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:29.694517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:29.694519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:29.694774 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:29.694984 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:29.694985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:29.694985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:29.695100 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:29.695403 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:29.695412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:29.695563 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.695656 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.695836 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260812 14:30:29.696439 15791 heartbeater.cc:347] Connected to a master server at 127.23.173.254:38695
I20260812 14:30:29.696533 15791 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:29.696696 15791 heartbeater.cc:511] Master 127.23.173.254:38695 requested a full tablet report, sending...
I20260812 14:30:29.697032 15004 ts_manager.cc:194] Registered new tserver with Master: 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
I20260812 14:30:29.697399 15004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:53813
I20260812 14:30:29.702206 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:30:29.712984 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.703409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.23.173.254:38695)
0812 14:30:29.703454 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.703456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.703554 (+ 98us) server_negotiation.cc:407] Connection header received
0812 14:30:29.703601 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.703602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.703610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.703656 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:29.704169 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.704616 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.705240 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.705403 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.705441 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.710943 (+ 5502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:29.710944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:29.710944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:29.710950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:29.712137 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.712431 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.712432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.712432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.712476 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:29.712613 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:29.712614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:29.712614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:29.712730 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:29.712799 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.712877 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.712928 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:30:29.714283 15004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:30:29.715782 15004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:30:29.721822 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.717852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.23.173.194:39383)
0812 14:30:29.718026 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.718031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.718087 (+ 56us) server_negotiation.cc:407] Connection header received
0812 14:30:29.718176 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.718178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.718193 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.718271 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:29.718545 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.719293 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.720629 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.721342 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.721391 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.721439 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.721700 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.721740 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260812 14:30:29.721834 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.717870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:46253 (local address 127.0.0.1:41240)
0812 14:30:29.718174 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.718182 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.718227 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.718372 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.718373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.718375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:29.718475 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.718477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.719784 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.719785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.720841 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.720843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.721585 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.721586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.721651 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.721656 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.721703 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.721764 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260812 14:30:29.721840 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.717938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.23.173.195:46253)
0812 14:30:29.718103 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.718106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.718237 (+ 131us) server_negotiation.cc:407] Connection header received
0812 14:30:29.718276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.718278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.718292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.718366 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:29.718560 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.719669 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.720911 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.721382 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.721583 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.721673 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.721723 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.721764 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260812 14:30:29.722270 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.717677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39383 (local address 127.0.0.1:43490)
0812 14:30:29.718029 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.718037 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.718092 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.718369 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.718372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.718374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:29.718443 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.718445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.719385 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.719386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.720520 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.720523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.721496 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.721496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.721590 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.721602 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.722176 (+ 574us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.722218 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260812 14:30:29.722517 15684 tablet_service.cc:1467] Processing CreateTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 (DEFAULT_TABLE table=test-table [id=03e37aa72e8b4482beedf9e1d019939e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:29.722880 15684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14eaae1ee7fc42d9826078bb4a995e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:29.723004 15453 tablet_service.cc:1467] Processing CreateTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 (DEFAULT_TABLE table=test-table [id=03e37aa72e8b4482beedf9e1d019939e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:29.723325 15453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14eaae1ee7fc42d9826078bb4a995e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:29.726145 15818 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Bootstrap starting.
I20260812 14:30:29.726641 15821 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Bootstrap starting.
I20260812 14:30:29.726823 15818 tablet_bootstrap.cc:654] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:29.727161 15821 tablet_bootstrap.cc:654] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:29.727402 15818 log.cc:824] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:29.727638 15821 log.cc:824] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:29.727820 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.717677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:38443 (local address 127.0.0.1:37496)
0812 14:30:29.717976 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.718025 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.718092 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.718464 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.718465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.718467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:29.718523 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.718525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.720928 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.720928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.722093 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.722096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.727545 (+ 5449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.727546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.727614 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.727618 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.727676 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.727739 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":384,"thread_start_us":66,"threads_started":1}
I20260812 14:30:29.728077 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.717764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.23.173.193:38443)
0812 14:30:29.718247 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.718252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.718262 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:30:29.718298 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.718300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.718313 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.718377 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:29.718602 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.719675 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.726412 (+ 6737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.727424 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.727536 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.727605 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.727934 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.727985 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":99,"threads_started":1}
I20260812 14:30:29.728340 15818 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: No bootstrap required, opened a new log
I20260812 14:30:29.728389 15821 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: No bootstrap required, opened a new log
I20260812 14:30:29.728405 15818 ts_tablet_manager.cc:1397] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:29.728448 15821 ts_tablet_manager.cc:1397] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:30:29.728749 15218 tablet_service.cc:1467] Processing CreateTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 (DEFAULT_TABLE table=test-table [id=03e37aa72e8b4482beedf9e1d019939e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:29.729058 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14eaae1ee7fc42d9826078bb4a995e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:29.729869 15818 raft_consensus.cc:348] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.729995 15818 raft_consensus.cc:374] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:29.730026 15818 raft_consensus.cc:729] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d9d5f039e841c394cebd17bd4b6f80, State: Initialized, Role: FOLLOWER
I20260812 14:30:29.730162 15818 consensus_queue.cc:260] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.730409 15818 ts_tablet_manager.cc:1428] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:29.730297 15821 raft_consensus.cc:348] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.730448 15821 raft_consensus.cc:374] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:29.730492 15821 raft_consensus.cc:729] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1a820dbaf84193b40fdf3cada07cfe, State: Initialized, Role: FOLLOWER
I20260812 14:30:29.730515 15791 heartbeater.cc:503] Master 127.23.173.254:38695 was elected leader, sending a full tablet report...
I20260812 14:30:29.730605 15821 consensus_queue.cc:260] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.730875 15821 ts_tablet_manager.cc:1428] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:30:29.731052 15568 heartbeater.cc:503] Master 127.23.173.254:38695 was elected leader, sending a full tablet report...
I20260812 14:30:29.732148 15831 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap starting.
I20260812 14:30:29.732760 15831 tablet_bootstrap.cc:654] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:29.733189 15831 log.cc:824] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:29.733765 15831 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: No bootstrap required, opened a new log
I20260812 14:30:29.733814 15831 ts_tablet_manager.cc:1397] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:30:29.735072 15831 raft_consensus.cc:348] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.735188 15831 raft_consensus.cc:374] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:29.735217 15831 raft_consensus.cc:729] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf5d3e39ec2447cadfa700ebbbfc84d, State: Initialized, Role: FOLLOWER
I20260812 14:30:29.735319 15831 consensus_queue.cc:260] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.735569 15831 ts_tablet_manager.cc:1428] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:29.735590 15327 heartbeater.cc:503] Master 127.23.173.254:38695 was elected leader, sending a full tablet report...
W20260812 14:30:29.783456 15570 tablet.cc:2369] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:30:29.891748 15328 tablet.cc:2369] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:30:29.937247 15792 tablet.cc:2369] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:30:29.980064 15827 raft_consensus.cc:482] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:29.980197 15827 raft_consensus.cc:504] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.980572 15827 leader_election.cc:288] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443)
I20260812 14:30:29.983261 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.980819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:57329 (local address 127.23.173.194:39383)
0812 14:30:29.980871 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.980873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.980884 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:30:29.980978 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.980979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.981000 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.981048 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:29.981195 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.981663 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.982562 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.982936 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.983011 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.983020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.983158 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.983194 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:30:29.983294 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.980763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39383 (local address 127.23.173.195:57329)
0812 14:30:29.980838 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.980849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.980893 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.981066 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.981067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.981069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:29.981137 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.981139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.981738 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.981739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.982465 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.982467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.983014 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.983015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.983095 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.983098 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.983146 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.983190 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 14:30:29.983309 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.980833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:60605 (local address 127.23.173.193:38443)
0812 14:30:29.980880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:29.980882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:29.981018 (+ 136us) server_negotiation.cc:407] Connection header received
0812 14:30:29.981073 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:29.981074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:29.981084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:29.981130 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:29.981272 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.981730 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.982640 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.982993 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.983064 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.983072 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:29.983203 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:30:29.983244 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:30:29.983435 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:29.980773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:38443 (local address 127.23.173.195:60605)
0812 14:30:29.980939 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:29.980949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:29.981008 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:29.981143 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:29.981143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:29.981145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:29.981210 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.981212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.981808 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.981809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.982553 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:29.982555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:29.983069 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:29.983070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:29.983137 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:29.983140 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:30:29.983185 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:30:29.983224 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260812 14:30:29.983685 15484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eaae1ee7fc42d9826078bb4a995e26" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" is_pre_election: true
I20260812 14:30:29.983857 15484 raft_consensus.cc:2454] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 0.
I20260812 14:30:29.983806 15250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eaae1ee7fc42d9826078bb4a995e26" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" is_pre_election: true
I20260812 14:30:29.983959 15250 raft_consensus.cc:2454] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 0.
I20260812 14:30:29.984107 15602 leader_election.cc:302] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a1a820dbaf84193b40fdf3cada07cfe, 90d9d5f039e841c394cebd17bd4b6f80; no voters:
I20260812 14:30:29.984249 15827 raft_consensus.cc:2781] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:30:29.984319 15827 raft_consensus.cc:482] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:30:29.984335 15827 raft_consensus.cc:3037] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:29.985052 15827 raft_consensus.cc:504] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.985174 15827 leader_election.cc:288] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 election: Requested vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443)
I20260812 14:30:29.985318 15484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eaae1ee7fc42d9826078bb4a995e26" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
I20260812 14:30:29.985369 15250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eaae1ee7fc42d9826078bb4a995e26" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
I20260812 14:30:29.985399 15484 raft_consensus.cc:3037] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:29.985436 15250 raft_consensus.cc:3037] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:29.986126 15250 raft_consensus.cc:2454] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 1.
I20260812 14:30:29.986147 15484 raft_consensus.cc:2454] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 1.
I20260812 14:30:29.986303 15600 leader_election.cc:302] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d, 90d9d5f039e841c394cebd17bd4b6f80; no voters:
I20260812 14:30:29.986414 15827 raft_consensus.cc:2781] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:29.986568 15827 raft_consensus.cc:686] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 LEADER]: Becoming Leader. State: Replica: 90d9d5f039e841c394cebd17bd4b6f80, State: Running, Role: LEADER
I20260812 14:30:29.986656 15827 consensus_queue.cc:237] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:29.987473 15003 catalog_manager.cc:5495] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "90d9d5f039e841c394cebd17bd4b6f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } health_report { overall_health: HEALTHY } } }
I20260812 14:30:30.004105 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.002001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.23.173.195:46253)
0812 14:30:30.002086 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.002088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.002095 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:30.002124 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.002124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.002140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.002196 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:30:30.002282 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.002735 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.003415 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.003600 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.003648 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.003826 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:30:30.003917 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:30:30.003978 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.003992 (+ 14us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.004008 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:30:30.005695 15250 raft_consensus.cc:1261] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Refusing update from remote peer 90d9d5f039e841c394cebd17bd4b6f80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:30.005697 15484 raft_consensus.cc:1261] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Refusing update from remote peer 90d9d5f039e841c394cebd17bd4b6f80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:30.006083 15840 consensus_queue.cc:1035] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:30.006211 15827 consensus_queue.cc:1035] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:30.008920 15848 mvcc.cc:205] Tried to move back new op lower bound from 7317688442900979712 to 7317688442825818112. Current Snapshot: MvccSnapshot[applied={T|T < 7317688442900979712}]
I20260812 14:30:30.008976 15845 mvcc.cc:205] Tried to move back new op lower bound from 7317688442900979712 to 7317688442825818112. Current Snapshot: MvccSnapshot[applied={T|T < 7317688442900979712}]
I20260812 14:30:30.010234 15004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:30.010316 15004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:30.010986 15004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Sending DeleteTablet for 3 replicas of tablet 14eaae1ee7fc42d9826078bb4a995e26
I20260812 14:30:30.011336 15453 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:43490
I20260812 14:30:30.011590 15864 tablet_replica.cc:330] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: stopping tablet replica
I20260812 14:30:30.011584 15218 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:37496
I20260812 14:30:30.011664 15864 raft_consensus.cc:2229] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:30:30.011762 15864 raft_consensus.cc:2258] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:30.011811 15684 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:41240
I20260812 14:30:30.012079 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 15102
I20260812 14:30:30.012199 15864 ts_tablet_manager.cc:1905] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:30.018608 15864 ts_tablet_manager.cc:1918] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:30.018666 15864 log.cc:1196] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/14eaae1ee7fc42d9826078bb4a995e26
I20260812 14:30:30.018829 15864 ts_tablet_manager.cc:1939] T 14eaae1ee7fc42d9826078bb4a995e26 P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting consensus metadata
I20260812 14:30:30.026821 14984 catalog_manager.cc:4841] TS 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383): tablet 14eaae1ee7fc42d9826078bb4a995e26 (table test-table [id=03e37aa72e8b4482beedf9e1d019939e]) successfully deleted
I20260812 14:30:30.012008 15870 tablet_replica.cc:330] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: stopping tablet replica
I20260812 14:30:30.036751 15870 raft_consensus.cc:2229] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:30:30.036962 15870 raft_consensus.cc:2258] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:30.038017 15870 ts_tablet_manager.cc:1905] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:30.039465 15870 ts_tablet_manager.cc:1918] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:30.039536 15870 log.cc:1196] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/14eaae1ee7fc42d9826078bb4a995e26
I20260812 14:30:30.039816 15870 ts_tablet_manager.cc:1939] T 14eaae1ee7fc42d9826078bb4a995e26 P 90d9d5f039e841c394cebd17bd4b6f80: Deleting consensus metadata
I20260812 14:30:30.040222 14982 catalog_manager.cc:4841] TS 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): tablet 14eaae1ee7fc42d9826078bb4a995e26 (table test-table [id=03e37aa72e8b4482beedf9e1d019939e]) successfully deleted
W20260812 14:30:30.042670 14982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:38443 (error 108)
I20260812 14:30:30.042867 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 15336
I20260812 14:30:30.044780 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.042994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:38443 (local address 127.0.0.1:37504)
0812 14:30:30.044703 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.044748 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1685}
W20260812 14:30:30.048903 14982 catalog_manager.cc:4584] TS 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14eaae1ee7fc42d9826078bb4a995e26: Network error: Client connection negotiation failed: client connection to 127.23.173.193:38443: connect: Connection refused (error 111)
I20260812 14:30:30.057173 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 15574
I20260812 14:30:30.069133 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 14948
I20260812 14:30:30.081202 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:38695
--webserver_interface=127.23.173.254
--webserver_port=38317
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:30.161573 15873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:30.161768 15873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:30.161790 15873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:30.162904 15873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:30.162946 15873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:30.162959 15873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:30.162982 15873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:30.163005 15873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:30.164476 15873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=38317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:30.164709 15873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:30.164924 15873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:30.174086 15882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.174130 15884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.174062 15881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:30.174290 15873 server_base.cc:1034] running on GCE node
I20260812 14:30:30.174515 15873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:30.174785 15873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:30.175923 15873 hybrid_clock.cc:648] HybridClock initialized: now 1786545030175908 us; error 30 us; skew 500 ppm
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:30.178332 15873 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:30.178719 15873 webserver.cc:460] Webserver started at http://127.23.173.254:38317/ using document root <none> and password file <none>
I20260812 14:30:30.178900 15873 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:30.178952 15873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:30.180291 15873 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:30.180893 15894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.181054 15873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:30.181143 15873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "33d9896a3e2047caa7da5b80a8fc5564"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "983f4baa0bf280936b0f2ea42f09b150"
server_key_iv: "aeff868a5a591d705187e745cefc3e13"
server_key_version: "encryptionkey@0"
I20260812 14:30:30.181509 15873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:30.206327 15873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:30.208400 15873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:30.208509 15873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:30.212323 15873 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:38695
I20260812 14:30:30.212378 15988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:38695 every 8 connection(s)
I20260812 14:30:30.212711 15873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:30.214326 15992 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:30:30.214983 15992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap starting.
I20260812 14:30:30.216161 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 15873
I20260812 14:30:30.216418 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:38443
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:38695
--webserver_port=40997
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:30.223657 15992 log.cc:824] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:30.225437 15992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:30:30.225697 15992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap complete.
I20260812 14:30:30.227803 15992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:30.228042 15992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d9896a3e2047caa7da5b80a8fc5564, State: Initialized, Role: FOLLOWER
I20260812 14:30:30.228191 15992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:30.228264 15992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:30.228317 15992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:30.228363 15992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:30:30.229329 15992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:30.229425 15992 leader_election.cc:302] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33d9896a3e2047caa7da5b80a8fc5564; no voters:
I20260812 14:30:30.229590 15992 leader_election.cc:288] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:30:30.229642 15996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:30:30.229755 15996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 LEADER]: Becoming Leader. State: Replica: 33d9896a3e2047caa7da5b80a8fc5564, State: Running, Role: LEADER
I20260812 14:30:30.229859 15992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:30.229924 15996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:30.230262 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d9896a3e2047caa7da5b80a8fc5564. Latest consensus state: current_term: 2 leader_uuid: "33d9896a3e2047caa7da5b80a8fc5564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } } }
I20260812 14:30:30.230350 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:30.230520 15997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "33d9896a3e2047caa7da5b80a8fc5564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } } }
I20260812 14:30:30.230654 15997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:30.230887 16005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:30.231827 16005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:30.231940 16005 catalog_manager.cc:1259] Loaded cluster ID: c5d2d369d61f491588621ac152fcaac7
I20260812 14:30:30.231973 16005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:30:30.232352 16005 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 14:30:30.237545 16005 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 14:30:30.239845 16005 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:30.240025 16005 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Loaded TSK: 0
I20260812 14:30:30.240228 16005 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:30:30.300293 15994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:30.300482 15994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:30.300504 15994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:30.301635 15994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:30.301676 15994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:30.301712 15994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:30:30.303192 15994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:38443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=40997
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:30.303412 15994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:30.303630 15994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:30.309448 16035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:30.309489 15994 server_base.cc:1034] running on GCE node
W20260812 14:30:30.309651 16037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.309476 16033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:30.309923 15994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:30.310160 15994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:30.311301 15994 hybrid_clock.cc:648] HybridClock initialized: now 1786545030311289 us; error 32 us; skew 500 ppm
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:30.313844 15994 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:30:30.314281 15994 webserver.cc:460] Webserver started at http://127.23.173.193:40997/ using document root <none> and password file <none>
I20260812 14:30:30.314472 15994 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:30.314544 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:30.315903 15994 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:30.316581 16046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.316736 15994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:30.316818 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "365a7921834e2f96bd0d610dc4d5defe"
server_key_iv: "02171e46183cd1f489b21af9d210ad19"
server_key_version: "encryptionkey@0"
I20260812 14:30:30.317137 15994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:30.339118 15994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:30.341030 15994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:30.341166 15994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:30.341367 15994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:30.341817 16056 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:30:30.343648 15994 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:30:30.343688 15994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:30:30.343739 15994 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:30:30.344331 15994 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:30:30.344362 15994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:30:30.344471 16056 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap starting.
I20260812 14:30:30.351127 15994 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:38443
I20260812 14:30:30.351285 16241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:38443 every 8 connection(s)
I20260812 14:30:30.351540 15994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:30:30.352524 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 15994
I20260812 14:30:30.352780 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:39383
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:38695
--webserver_port=43055
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:30.355556 16056 log.cc:824] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:30.357703 16056 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:30:30.357999 16056 tablet_bootstrap.cc:492] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap complete.
I20260812 14:30:30.359061 16056 ts_tablet_manager.cc:1397] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260812 14:30:30.360327 16056 raft_consensus.cc:348] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:30.360637 16056 raft_consensus.cc:729] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf5d3e39ec2447cadfa700ebbbfc84d, State: Initialized, Role: FOLLOWER
I20260812 14:30:30.360795 16056 consensus_queue.cc:260] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } }
I20260812 14:30:30.361068 16056 ts_tablet_manager.cc:1428] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:30.366923 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.352150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:57617)
0812 14:30:30.355844 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.355888 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.356485 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.357239 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.357244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.357591 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:30.357867 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.357873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.359528 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.359529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.360749 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.360752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.361538 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.361539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.361639 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.362169 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:30.362177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:30.363629 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:30.365592 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:30.365599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:30.365600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:30.365798 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:30.366039 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:30.366039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:30.366039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:30.366125 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:30.366503 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:30.366506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:30.366605 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.366678 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.366788 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3651,"thread_start_us":110,"threads_started":1}
I20260812 14:30:30.367051 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.352226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:57617 (local address 127.23.173.254:38695)
0812 14:30:30.356810 (+ 4584us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.356819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.356855 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:30:30.356981 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.356987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.357038 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.357195 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:30.358009 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.359424 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.360832 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.361448 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.361572 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.363758 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:30.363763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:30.363763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:30.363775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:30.365508 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.365869 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.365872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.365872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.365981 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.366175 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.366176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.366176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.366445 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:30.366523 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.366738 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.366869 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4531,"thread_start_us":93,"threads_started":1}
I20260812 14:30:30.367574 16242 heartbeater.cc:347] Connected to a master server at 127.23.173.254:38695
I20260812 14:30:30.367690 16242 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:30.367900 16242 heartbeater.cc:511] Master 127.23.173.254:38695 requested a full tablet report, sending...
I20260812 14:30:30.368649 15916 ts_manager.cc:194] Registered new tserver with Master: 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443)
I20260812 14:30:30.369831 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:57617
I20260812 14:30:30.371135 16242 heartbeater.cc:503] Master 127.23.173.254:38695 was elected leader, sending a full tablet report...
I20260812 14:30:30.375012 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.370449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.23.173.193:38443)
0812 14:30:30.371761 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.371766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.371774 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:30:30.371802 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.371804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.371815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.371884 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:30.372057 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.372763 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.374216 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.374664 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.374726 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.374758 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.374924 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.374957 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":110,"threads_started":1}
I20260812 14:30:30.375041 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.370218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:38443 (local address 127.0.0.1:37514)
0812 14:30:30.370544 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.370579 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.371275 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.371913 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.371917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.371920 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:30.371987 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.371989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.372834 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.372835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.374120 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.374123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.374740 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.374741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.374827 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.374841 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.374905 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.374966 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260812 14:30:30.375420 16125 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eaae1ee7fc42d9826078bb4a995e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:37514
I20260812 14:30:30.375901 16253 tablet_replica.cc:330] stopping tablet replica
I20260812 14:30:30.376008 16253 raft_consensus.cc:2229] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:30:30.376070 16253 raft_consensus.cc:2258] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:30.376374 16253 ts_tablet_manager.cc:1905] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:30.378662 16253 ts_tablet_manager.cc:1918] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:30.378738 16253 log.cc:1196] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/14eaae1ee7fc42d9826078bb4a995e26
I20260812 14:30:30.378985 16253 ts_tablet_manager.cc:1939] T 14eaae1ee7fc42d9826078bb4a995e26 P 0cf5d3e39ec2447cadfa700ebbbfc84d: Deleting consensus metadata
I20260812 14:30:30.379442 15895 catalog_manager.cc:4841] TS 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443): tablet 14eaae1ee7fc42d9826078bb4a995e26 (table test-table [id=03e37aa72e8b4482beedf9e1d019939e]) successfully deleted
W20260812 14:30:30.440866 16246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:30.441056 16246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:30.441079 16246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:30.442173 16246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:30.442209 16246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:30.442233 16246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:30:30.443607 16246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:39383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=43055
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:30.443819 16246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:30.444018 16246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:30.448602 16261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.448722 16260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.448776 16264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:30.448791 16246 server_base.cc:1034] running on GCE node
I20260812 14:30:30.449077 16246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:30.449270 16246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:30.450439 16246 hybrid_clock.cc:648] HybridClock initialized: now 1786545030450424 us; error 30 us; skew 500 ppm
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:30.452867 16246 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:30:30.453289 16246 webserver.cc:460] Webserver started at http://127.23.173.194:43055/ using document root <none> and password file <none>
I20260812 14:30:30.453510 16246 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:30.453573 16246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:30.454813 16246 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:30:30.455314 16276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.455464 16246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:30.455550 16246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "d5ff9cbb51dd6042e11d393b25395a1c"
server_key_iv: "b99927244ac306c6488dd03655dcdf52"
server_key_version: "encryptionkey@0"
I20260812 14:30:30.455897 16246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:30.475292 16246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:30.476938 16246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:30.477066 16246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:30.477269 16246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:30.477550 16246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:30.477581 16246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.477609 16246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:30.477627 16246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.483119 16246 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:39383
I20260812 14:30:30.483155 16503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:39383 every 8 connection(s)
I20260812 14:30:30.483479 16246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:30.489140 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 16246
I20260812 14:30:30.489373 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:46253
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:38695
--webserver_port=32927
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:30.491946 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.484169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:49075 (local address 127.23.173.254:38695)
0812 14:30:30.484238 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.484240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.484423 (+ 183us) server_negotiation.cc:407] Connection header received
0812 14:30:30.484889 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.484889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.484902 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.484948 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:30.485792 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.486281 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.487220 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.487428 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.487478 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.489377 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:30.489379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:30.489379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:30.489386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:30.490666 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.491034 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.491035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.491036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.491099 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.491344 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.491344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.491345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.491473 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:30.491537 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.491760 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.491849 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:30:30.492023 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.484142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:49075)
0812 14:30:30.484307 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.484341 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.484801 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.485001 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.485004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.485547 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:30.485722 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.485726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.486390 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.486391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.487156 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.487159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.487490 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.487491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.487584 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.487985 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:30.487992 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:30.489287 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:30.490746 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:30.490753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:30.490755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:30.490963 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:30.491173 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:30.491174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:30.491175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:30.491282 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:30.491530 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:30.491533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:30.491634 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.491747 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.491869 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260812 14:30:30.492456 16504 heartbeater.cc:347] Connected to a master server at 127.23.173.254:38695
I20260812 14:30:30.492549 16504 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:30.492717 16504 heartbeater.cc:511] Master 127.23.173.254:38695 requested a full tablet report, sending...
I20260812 14:30:30.493076 15916 ts_manager.cc:194] Registered new tserver with Master: 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383)
I20260812 14:30:30.493417 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:49075
W20260812 14:30:30.571398 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:30.571575 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:30.571605 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:30.572731 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:30.572774 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:30.572808 16508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:30:30.574254 16508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:46253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=32927
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:30.574482 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:30.574684 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:30.579433 16527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.579429 16523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:30.579407 16522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:30.579668 16508 server_base.cc:1034] running on GCE node
I20260812 14:30:30.579895 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:30.580125 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:30.581265 16508 hybrid_clock.cc:648] HybridClock initialized: now 1786545030581248 us; error 34 us; skew 500 ppm
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:30.583592 16508 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:30:30.584080 16508 webserver.cc:460] Webserver started at http://127.23.173.195:32927/ using document root <none> and password file <none>
I20260812 14:30:30.584295 16508 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:30.584350 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:30.585619 16508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:30:30.586205 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.586356 16508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:30.586429 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "90d9d5f039e841c394cebd17bd4b6f80"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "a65e1f3c7df8b65243a3f9b581047ac4"
server_key_iv: "403e688784545e5a6f950b5f19daca98"
server_key_version: "encryptionkey@0"
I20260812 14:30:30.586732 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:30.602317 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:30.604099 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:30.604216 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:30.604413 16508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:30.604682 16508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:30.604715 16508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.604735 16508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:30.604758 16508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.610870 16508 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:46253
I20260812 14:30:30.610920 16722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:46253 every 8 connection(s)
I20260812 14:30:30.611210 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:30:30.615897 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 16508
Aug 12 14:30:30 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:30.620723 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.611918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:43851 (local address 127.23.173.254:38695)
0812 14:30:30.611962 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.611964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.612162 (+ 198us) server_negotiation.cc:407] Connection header received
0812 14:30:30.612669 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.612670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.612682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.612735 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:30.613635 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.614122 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.615073 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.615267 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.615323 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.618030 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:30.618031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:30.618032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:30.618040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:30.619307 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.619690 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.619691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.619691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.619791 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.620085 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.620086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.620087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.620224 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:30.620275 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.620588 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.620659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:30:30.620949 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.611897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:43851)
0812 14:30:30.612056 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.612089 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.612573 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.612800 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.612804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.613356 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:30.613543 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.613547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.614204 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.614205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.614990 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.614992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.615349 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.615350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.615472 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.615953 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:30.615964 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:30.617941 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:30.619381 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:30.619388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:30.619391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:30.619607 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:30.619851 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:30.619852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:30.619852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:30.620016 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:30.620283 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:30.620291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:30.620445 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.620563 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.620722 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260812 14:30:30.621443 16731 heartbeater.cc:347] Connected to a master server at 127.23.173.254:38695
I20260812 14:30:30.621547 16731 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:30.621747 16731 heartbeater.cc:511] Master 127.23.173.254:38695 requested a full tablet report, sending...
I20260812 14:30:30.622161 15916 ts_manager.cc:194] Registered new tserver with Master: 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
I20260812 14:30:30.622243 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.616276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.23.173.254:38695)
0812 14:30:30.616393 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.616395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.616425 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:30:30.616490 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.616491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.616503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.616561 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:30.617097 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.617584 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.618268 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.618505 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.618576 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.619657 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:30.619659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:30.619659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:30.619665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:30.621120 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.621445 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.621446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.621446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.621507 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:30.621691 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:30.621691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:30.621692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:30.621818 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:30.622001 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.622040 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.622120 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260812 14:30:30.622617 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:43851
I20260812 14:30:30.622962 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:30:30.625499 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.623262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.23.173.254:38695)
0812 14:30:30.623294 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.623295 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.623403 (+ 108us) server_negotiation.cc:407] Connection header received
0812 14:30:30.623480 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.623481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.623523 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.623581 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:30:30.623672 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.624136 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.624812 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.625015 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.625068 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.625214 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:30:30.625308 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:30:30.625374 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.625410 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.625431 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:30:30.625789 15915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:30:30.627739 15915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:30:30.629901 16125 tablet_service.cc:1467] Processing CreateTablet for tablet d4ad73836dec43a3aa189e4ec26f7f8b (DEFAULT_TABLE table=test-table [id=81922fa159b5436d94e4e8bbd110221a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:30.630106 16125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ad73836dec43a3aa189e4ec26f7f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:30.631831 16056 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap starting.
I20260812 14:30:30.632156 16056 tablet_bootstrap.cc:654] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:30.632591 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.629374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:46253 (local address 127.0.0.1:41248)
0812 14:30:30.629573 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.629582 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.629614 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.629773 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.629774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.629775 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:30.629810 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.629812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.630865 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.630865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.631533 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.631535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.632336 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.632337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.632421 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.632424 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.632478 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.632521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":12,"thread_start_us":93,"threads_started":1}
I20260812 14:30:30.632773 16056 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: No bootstrap required, opened a new log
I20260812 14:30:30.632824 16056 ts_tablet_manager.cc:1397] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:30:30.632870 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.629468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.23.173.195:46253)
0812 14:30:30.629647 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.629651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.629660 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:30.629690 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.629692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.629706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.629776 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:30.629858 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.630748 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.631607 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.632266 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.632341 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.632396 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.632760 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.632798 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260812 14:30:30.632925 16056 raft_consensus.cc:348] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.633006 16056 raft_consensus.cc:374] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:30.633054 16056 raft_consensus.cc:729] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf5d3e39ec2447cadfa700ebbbfc84d, State: Initialized, Role: FOLLOWER
I20260812 14:30:30.633104 16056 consensus_queue.cc:260] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.633191 16056 ts_tablet_manager.cc:1428] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:30.633572 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.629365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39383 (local address 127.0.0.1:43506)
0812 14:30:30.629533 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.629544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.629594 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.630012 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.630013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.630015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:30.630048 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.630049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.631255 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.631256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.632450 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.632452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.633344 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.633345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.633415 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.633418 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.633470 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.633508 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260812 14:30:30.633594 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.629474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.23.173.194:39383)
0812 14:30:30.629852 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.629857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.629867 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:30:30.629905 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.629907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.629922 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.630002 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:30.630108 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.631133 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.632529 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.633273 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.633344 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.633396 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.633484 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.633520 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":72,"threads_started":1}
I20260812 14:30:30.633525 16622 tablet_service.cc:1467] Processing CreateTablet for tablet d4ad73836dec43a3aa189e4ec26f7f8b (DEFAULT_TABLE table=test-table [id=81922fa159b5436d94e4e8bbd110221a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:30.633848 16622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ad73836dec43a3aa189e4ec26f7f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:30.634249 16384 tablet_service.cc:1467] Processing CreateTablet for tablet d4ad73836dec43a3aa189e4ec26f7f8b (DEFAULT_TABLE table=test-table [id=81922fa159b5436d94e4e8bbd110221a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:30.634518 16384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ad73836dec43a3aa189e4ec26f7f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:30.636543 16748 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Bootstrap starting.
I20260812 14:30:30.637296 16748 tablet_bootstrap.cc:654] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:30.637356 16749 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Bootstrap starting.
I20260812 14:30:30.637761 16748 log.cc:824] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:30.637882 16749 tablet_bootstrap.cc:654] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:30.638298 16749 log.cc:824] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:30.638415 16748 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: No bootstrap required, opened a new log
I20260812 14:30:30.638484 16748 ts_tablet_manager.cc:1397] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:30:30.638957 16749 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: No bootstrap required, opened a new log
I20260812 14:30:30.639030 16749 ts_tablet_manager.cc:1397] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:30:30.640084 16748 raft_consensus.cc:348] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.640209 16748 raft_consensus.cc:374] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:30.640229 16748 raft_consensus.cc:729] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d9d5f039e841c394cebd17bd4b6f80, State: Initialized, Role: FOLLOWER
I20260812 14:30:30.640322 16748 consensus_queue.cc:260] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.640568 16748 ts_tablet_manager.cc:1428] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:30.640614 16731 heartbeater.cc:503] Master 127.23.173.254:38695 was elected leader, sending a full tablet report...
I20260812 14:30:30.640707 16749 raft_consensus.cc:348] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.640827 16749 raft_consensus.cc:374] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:30.640864 16749 raft_consensus.cc:729] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1a820dbaf84193b40fdf3cada07cfe, State: Initialized, Role: FOLLOWER
I20260812 14:30:30.640964 16749 consensus_queue.cc:260] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.641232 16749 ts_tablet_manager.cc:1428] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:30.641245 16504 heartbeater.cc:503] Master 127.23.173.254:38695 was elected leader, sending a full tablet report...
W20260812 14:30:30.734045 16506 tablet.cc:2369] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:30:30.857056 16243 tablet.cc:2369] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:30:30.861743 16732 tablet.cc:2369] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:30:30.931187 16752 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:30.931319 16752 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.931722 16752 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443)
I20260812 14:30:30.934439 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.931988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:50731 (local address 127.23.173.193:38443)
0812 14:30:30.932119 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.932121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.932128 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:30.932153 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.932153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.932162 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.932218 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:30.932390 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.932858 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.933739 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.934108 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.934171 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.934180 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.934318 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.934367 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260812 14:30:30.934439 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.931913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:38443 (local address 127.23.173.195:50731)
0812 14:30:30.931970 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.931982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.932030 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.932232 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.932233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.932234 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:30.932300 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.932302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.932937 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.932938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.933652 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.933654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.934180 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.934181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.934257 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.934259 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.934303 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.934345 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:30:30.934469 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.931982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:47781 (local address 127.23.173.194:39383)
0812 14:30:30.932028 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.932030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.932203 (+ 173us) server_negotiation.cc:407] Connection header received
0812 14:30:30.932251 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.932252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.932261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.932305 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:30.932438 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.932903 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.933789 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.934163 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.934223 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.934232 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.934364 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.934406 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:30:30.934707 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.931916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39383 (local address 127.23.173.195:47781)
0812 14:30:30.932127 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:30.932136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:30.932195 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:30.932308 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:30.932308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:30.932309 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:30.932380 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.932382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.932979 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.932980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.933707 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:30.933708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.934240 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.934241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:30.934302 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.934305 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:30:30.934349 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:30:30.934386 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":81,"thread_start_us":75,"threads_started":1}
I20260812 14:30:30.934760 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ad73836dec43a3aa189e4ec26f7f8b" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" is_pre_election: true
I20260812 14:30:30.934918 16155 raft_consensus.cc:2454] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 0.
I20260812 14:30:30.935035 16416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ad73836dec43a3aa189e4ec26f7f8b" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" is_pre_election: true
I20260812 14:30:30.935154 16544 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d, 90d9d5f039e841c394cebd17bd4b6f80; no voters:
I20260812 14:30:30.935186 16416 raft_consensus.cc:2454] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 0.
I20260812 14:30:30.935276 16752 raft_consensus.cc:2781] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:30:30.935364 16752 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:30:30.935400 16752 raft_consensus.cc:3037] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:30.936131 16752 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.936249 16752 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 election: Requested vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443)
I20260812 14:30:30.936386 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ad73836dec43a3aa189e4ec26f7f8b" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
I20260812 14:30:30.936456 16416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ad73836dec43a3aa189e4ec26f7f8b" candidate_uuid: "90d9d5f039e841c394cebd17bd4b6f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
I20260812 14:30:30.936474 16155 raft_consensus.cc:3037] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:30.936547 16416 raft_consensus.cc:3037] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:30.937315 16155 raft_consensus.cc:2454] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 1.
I20260812 14:30:30.937336 16416 raft_consensus.cc:2454] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d9d5f039e841c394cebd17bd4b6f80 in term 1.
I20260812 14:30:30.937482 16544 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d, 90d9d5f039e841c394cebd17bd4b6f80; no voters:
I20260812 14:30:30.937583 16752 raft_consensus.cc:2781] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:30.937749 16752 raft_consensus.cc:686] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 LEADER]: Becoming Leader. State: Replica: 90d9d5f039e841c394cebd17bd4b6f80, State: Running, Role: LEADER
I20260812 14:30:30.937887 16752 consensus_queue.cc:237] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:30.938655 15915 catalog_manager.cc:5495] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "90d9d5f039e841c394cebd17bd4b6f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260812 14:30:30.984905 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:30.982652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.23.173.195:46253)
0812 14:30:30.982708 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:30.982710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:30.982719 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:30.982786 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:30.982787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:30.982803 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:30.982856 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:30:30.982967 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.983430 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.984175 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:30.984398 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:30.984466 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:30.984599 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:30:30.984708 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:30:30.984783 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:30.984795 (+ 12us) server_negotiation.cc:299] Negotiation successful
0812 14:30:30.984813 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:30:30.986574 16416 raft_consensus.cc:1261] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Refusing update from remote peer 90d9d5f039e841c394cebd17bd4b6f80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:30.986594 16155 raft_consensus.cc:1261] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Refusing update from remote peer 90d9d5f039e841c394cebd17bd4b6f80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:30.986912 16762 consensus_queue.cc:1035] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:30.987035 16752 consensus_queue.cc:1035] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:30.989621 16772 mvcc.cc:205] Tried to move back new op lower bound from 7317688446918639616 to 7317688446722023424. Current Snapshot: MvccSnapshot[applied={T|T < 7317688446918639616}]
I20260812 14:30:30.991624 15915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:30.991724 15915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:30.992383 15915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Sending DeleteTablet for 3 replicas of tablet d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:30.992700 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 15994
I20260812 14:30:30.992697 16384 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ad73836dec43a3aa189e4ec26f7f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:43506
I20260812 14:30:30.992736 16125 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ad73836dec43a3aa189e4ec26f7f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:37514
I20260812 14:30:30.992763 16622 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ad73836dec43a3aa189e4ec26f7f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:30 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:41248
I20260812 14:30:31.007952 16784 tablet_replica.cc:330] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: stopping tablet replica
I20260812 14:30:31.008085 16784 raft_consensus.cc:2229] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:30:31.008217 16784 raft_consensus.cc:2258] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:31.008638 16784 ts_tablet_manager.cc:1905] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:31.010161 16784 ts_tablet_manager.cc:1918] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:31.010236 16784 log.cc:1196] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:31.010476 16784 ts_tablet_manager.cc:1939] T d4ad73836dec43a3aa189e4ec26f7f8b P 90d9d5f039e841c394cebd17bd4b6f80: Deleting consensus metadata
I20260812 14:30:31.010883 15895 catalog_manager.cc:4841] TS 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): tablet d4ad73836dec43a3aa189e4ec26f7f8b (table test-table [id=81922fa159b5436d94e4e8bbd110221a]) successfully deleted
I20260812 14:30:31.011816 16781 tablet_replica.cc:330] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: stopping tablet replica
I20260812 14:30:31.011938 16781 raft_consensus.cc:2229] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:30:31.012071 16781 raft_consensus.cc:2258] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:31.012465 16781 ts_tablet_manager.cc:1905] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:30:31.012692 15895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:38443 (error 108)
I20260812 14:30:31.013258 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.013124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:38443 (local address 127.0.0.1:37520)
0812 14:30:31.013175 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.013221 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:30:31.013437 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 16246
W20260812 14:30:31.019745 15895 catalog_manager.cc:4584] TS 0cf5d3e39ec2447cadfa700ebbbfc84d (127.23.173.193:38443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4ad73836dec43a3aa189e4ec26f7f8b: Network error: Client connection negotiation failed: client connection to 127.23.173.193:38443: connect: Connection refused (error 111)
I20260812 14:30:31.023339 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 16508
I20260812 14:30:31.023370 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.023241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39383 (local address 127.0.0.1:43510)
0812 14:30:31.023301 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.023336 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.194:39383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:30:31.035917 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 15873
I20260812 14:30:31.045853 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:38695
--webserver_interface=127.23.173.254
--webserver_port=38317
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:31.126693 16786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:31.126891 16786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:31.126914 16786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:31.128075 16786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:31.128115 16786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:31.128129 16786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:31.128139 16786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:31.128149 16786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:31.129602 16786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=38317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:31.129817 16786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:31.130028 16786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:30:31.134724 16786 server_base.cc:1034] running on GCE node
W20260812 14:30:31.134696 16793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.134681 16802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.134671 16798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:31.135135 16786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:31.135412 16786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:31.136547 16786 hybrid_clock.cc:648] HybridClock initialized: now 1786545031136539 us; error 29 us; skew 500 ppm
Aug 12 14:30:31 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:31.138892 16786 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:31.139295 16786 webserver.cc:460] Webserver started at http://127.23.173.254:38317/ using document root <none> and password file <none>
I20260812 14:30:31.139484 16786 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:31.139533 16786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:31.140933 16786 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:31.141563 16814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.141706 16786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:31.141785 16786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "33d9896a3e2047caa7da5b80a8fc5564"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "983f4baa0bf280936b0f2ea42f09b150"
server_key_iv: "aeff868a5a591d705187e745cefc3e13"
server_key_version: "encryptionkey@0"
I20260812 14:30:31.142114 16786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:31.157136 16786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:31.159184 16786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:31.159322 16786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:31.163172 16786 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:38695
I20260812 14:30:31.163225 16919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:38695 every 8 connection(s)
I20260812 14:30:31.163565 16786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:31.165083 16920 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:30:31.165665 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap starting.
I20260812 14:30:31.170667 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 16786
I20260812 14:30:31.170909 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:38443
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:38695
--webserver_port=40997
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:31.173866 16920 log.cc:824] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:31.176015 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:30:31.176273 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564: Bootstrap complete.
I20260812 14:30:31.178334 16920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:31.178550 16920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d9896a3e2047caa7da5b80a8fc5564, State: Initialized, Role: FOLLOWER
I20260812 14:30:31.178691 16920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:31.178750 16920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:31.178788 16920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:31.178835 16920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:30:31.179831 16920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:31.179921 16920 leader_election.cc:302] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33d9896a3e2047caa7da5b80a8fc5564; no voters:
I20260812 14:30:31.180083 16920 leader_election.cc:288] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 14:30:31.180132 16927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:30:31.180245 16927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [term 3 LEADER]: Becoming Leader. State: Replica: 33d9896a3e2047caa7da5b80a8fc5564, State: Running, Role: LEADER
I20260812 14:30:31.180352 16920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:31.180339 16927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } }
I20260812 14:30:31.180650 16927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d9896a3e2047caa7da5b80a8fc5564. Latest consensus state: current_term: 3 leader_uuid: "33d9896a3e2047caa7da5b80a8fc5564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } } }
I20260812 14:30:31.180740 16927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:31.180645 16931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "33d9896a3e2047caa7da5b80a8fc5564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d9896a3e2047caa7da5b80a8fc5564" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 38695 } } }
I20260812 14:30:31.180910 16931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d9896a3e2047caa7da5b80a8fc5564 [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:31.180940 16938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:31.181771 16938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:31.181879 16938 catalog_manager.cc:1259] Loaded cluster ID: c5d2d369d61f491588621ac152fcaac7
I20260812 14:30:31.181906 16938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 14:30:31.182202 16938 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbb0c5ad9ad google::LogMessage::Fail() at ??:0
@ 0x7fbb0c5b088a google::LogMessage::SendToLog() at ??:0
@ 0x7fbb0c5ad4de google::LogMessage::Flush() at ??:0
@ 0x7fbb0c5aea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbb144ce0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbb144d78b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbb0e47b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbb0e470954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbb0f2d46db start_thread at ??:0
@ 0x7fbb0b0ab61f clone at ??:0
W20260812 14:30:31.266995 16925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:31.267189 16925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:31.267212 16925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:31.268383 16925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:31.268425 16925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:31.268450 16925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:30:31.269893 16925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:38443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=40997
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:31.270114 16925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:31.270330 16925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:31.276208 16982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.276268 16988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.276247 16983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:31.276610 16925 server_base.cc:1034] running on GCE node
I20260812 14:30:31.276785 16925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:31.277014 16925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:31.278190 16925 hybrid_clock.cc:648] HybridClock initialized: now 1786545031278171 us; error 34 us; skew 500 ppm
Aug 12 14:30:31 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:31.280556 16925 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:30:31.280977 16925 webserver.cc:460] Webserver started at http://127.23.173.193:40997/ using document root <none> and password file <none>
I20260812 14:30:31.281212 16925 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:31.281256 16925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:31.282506 16925 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:30:31.283134 17002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.283320 16925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:31.283439 16925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "365a7921834e2f96bd0d610dc4d5defe"
server_key_iv: "02171e46183cd1f489b21af9d210ad19"
server_key_version: "encryptionkey@0"
I20260812 14:30:31.283911 16925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:31.308063 16925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:31.310374 16925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:31.310539 16925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:31.310822 16925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:31.311326 17015 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:30:31.312772 16925 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:30:31.312834 16925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:30:31.312870 16925 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:30:31.313781 16925 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:30:31.313825 16925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:30:31.313908 17015 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap starting.
I20260812 14:30:31.321257 16925 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:38443
I20260812 14:30:31.321516 17209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:38443 every 8 connection(s)
I20260812 14:30:31.321733 16925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:30:31.324478 17015 log.cc:824] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:31.326726 17015 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:30:31.326822 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 16925
I20260812 14:30:31.327029 17015 tablet_bootstrap.cc:492] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Bootstrap complete.
I20260812 14:30:31.327119 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:39383
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:38695
--webserver_port=43055
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:30:31.328188 17015 ts_tablet_manager.cc:1397] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.012s
I20260812 14:30:31.329371 17015 raft_consensus.cc:348] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:31.329604 17015 raft_consensus.cc:729] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf5d3e39ec2447cadfa700ebbbfc84d, State: Initialized, Role: FOLLOWER
I20260812 14:30:31.329730 17015 consensus_queue.cc:260] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:31.329964 17015 ts_tablet_manager.cc:1428] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260812 14:30:31.425942 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:31.426186 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:31.426213 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:31.427359 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:31.427399 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:31.427423 17217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
W20260812 14:30:31.428498 17213 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:30:31.322474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:43537)
0812 14:30:31.322698 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.322749 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:31.323356 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:31.428302 (+104946us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
W20260812 14:30:31.428714 17210 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:38695 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.254:38695: Client connection negotiation failed: client connection to 127.23.173.254:38695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:30:31.428994 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:39383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=43055
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:31.429075 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.428965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:52651)
0812 14:30:31.429013 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.429054 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:30:31.429229 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:31.429453 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:30:31.429468 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.429348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:41541)
0812 14:30:31.429401 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.429441 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:30:31.429589 17210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:30:31.433995 17233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.434006 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.434118 17226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:31.434403 17217 server_base.cc:1034] running on GCE node
I20260812 14:30:31.434597 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:31.434821 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:31.435957 17217 hybrid_clock.cc:648] HybridClock initialized: now 1786545031435940 us; error 27 us; skew 500 ppm
Aug 12 14:30:31 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:31.438239 17217 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:30:31.438676 17217 webserver.cc:460] Webserver started at http://127.23.173.194:43055/ using document root <none> and password file <none>
I20260812 14:30:31.438881 17217 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:31.438931 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:31.440246 17217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:30:31.440933 17249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.441077 17217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:31.441155 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "3a1a820dbaf84193b40fdf3cada07cfe"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "d5ff9cbb51dd6042e11d393b25395a1c"
server_key_iv: "b99927244ac306c6488dd03655dcdf52"
server_key_version: "encryptionkey@0"
I20260812 14:30:31.441501 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:31.455535 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:31.457525 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:31.457653 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:31.457841 17217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:31.458204 17261 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260812 14:30:31.459558 17261 ts_tablet_manager.cc:1852] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 14:30:31.459944 17261 ts_tablet_manager.cc:1905] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:31.461962 17261 ts_tablet_manager.cc:1918] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:31.462045 17261 log.cc:1196] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:31.462330 17261 ts_tablet_manager.cc:1939] T d4ad73836dec43a3aa189e4ec26f7f8b P 3a1a820dbaf84193b40fdf3cada07cfe: Deleting consensus metadata
I20260812 14:30:31.462518 17217 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 14:30:31.462566 17217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260812 14:30:31.462604 17217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:31.462630 17217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.468315 17217 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:39383
I20260812 14:30:31.468376 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:39383 every 8 connection(s)
I20260812 14:30:31.468641 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:30:31.469532 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.469202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:59583)
0812 14:30:31.469343 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.469408 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260812 14:30:31.469696 17462 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:38695 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.254:38695: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
I20260812 14:30:31.469924 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.469850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:41995)
0812 14:30:31.469885 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.469908 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:30:31.470121 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.470076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:36435)
0812 14:30:31.470094 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.470110 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260812 14:30:31.470184 17462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:30:31.475077 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 17217
I20260812 14:30:31.475313 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:46253
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:38695
--webserver_port=32927
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:31.556372 17467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:31.556561 17467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:31.556583 17467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:31.557704 17467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:31.557745 17467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:31.557773 17467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:30:31.559170 17467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:46253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=32927
--tserver_master_addrs=127.23.173.254:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:31.559393 17467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:31.559609 17467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:31.564376 17474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.564394 17476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:31.564410 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:31.564492 17467 server_base.cc:1034] running on GCE node
I20260812 14:30:31.564774 17467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:31.565021 17467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:31.566157 17467 hybrid_clock.cc:648] HybridClock initialized: now 1786545031566147 us; error 34 us; skew 500 ppm
Aug 12 14:30:31 dist-test-slave-1l3l krb5kdc[14926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:31.568465 17467 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:30:31.568856 17467 webserver.cc:460] Webserver started at http://127.23.173.195:32927/ using document root <none> and password file <none>
I20260812 14:30:31.569047 17467 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:31.569096 17467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:31.570333 17467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:30:31.570837 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.570993 17467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:30:31.571072 17467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "90d9d5f039e841c394cebd17bd4b6f80"
format_stamp: "Formatted at 2026-08-12 14:30:29 on dist-test-slave-1l3l"
server_key: "a65e1f3c7df8b65243a3f9b581047ac4"
server_key_iv: "403e688784545e5a6f950b5f19daca98"
server_key_version: "encryptionkey@0"
I20260812 14:30:31.571368 17467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:31.588759 17467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:31.590510 17467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:31.590636 17467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:31.590835 17467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:31.591109 17467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:31.591142 17467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.591169 17467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:31.591194 17467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:31.596653 17467 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:46253
I20260812 14:30:31.596711 17685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:46253 every 8 connection(s)
I20260812 14:30:31.597013 17467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:30:31.597913 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.597566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:47811)
0812 14:30:31.597704 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.597777 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260812 14:30:31.598292 17690 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:38695 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.254:38695: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
I20260812 14:30:31.598546 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.598453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:51679)
0812 14:30:31.598500 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.598525 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:30:31.598776 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:31.598702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:33089)
0812 14:30:31.598736 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:31.598758 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 14:30:31.598865 17690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:30:31.600493 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 17467
I20260812 14:30:32.430153 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.429904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:43949)
0812 14:30:32.430041 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:32.430076 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260812 14:30:32.470649 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.470459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:44979)
0812 14:30:32.470579 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:32.470613 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260812 14:30:32.599340 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.599150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:42423)
0812 14:30:32.599258 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:32.599289 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260812 14:30:32.608903 17701 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:32.609005 17701 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:32.609323 17701 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
Aug 12 14:30:32 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.194@KRBTEST.COM
Aug 12 14:30:32 dist-test-slave-1l3l krb5kdc[14926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.195@KRBTEST.COM
I20260812 14:30:32.620806 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.609562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:38055 (local address 127.23.173.195:46253)
0812 14:30:32.609702 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:32.609714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:32.609748 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:30:32.610178 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:32.610180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:32.610221 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:32.610329 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:32.611132 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.612172 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.613599 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.614231 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.614702 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:32.617252 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:32.617256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:32.617256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:32.617267 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:32.619001 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619477 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:32.619480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:32.619481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:32.619624 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619974 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:32.619975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:32.619975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:32.620211 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:32.620284 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:32.620580 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 14:30:32.620694 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":47,"threads_started":1}
I20260812 14:30:32.620790 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.609522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39383 (local address 127.23.173.193:42713)
0812 14:30:32.609651 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:32.609659 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:32.609697 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:32.610442 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:32.610443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:32.610833 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:32.611016 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:32.611021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.612285 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.612286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:32.613498 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:32.613502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.614301 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.614303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:32.614438 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:32.615074 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:32.615084 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:32.616901 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:32.618989 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:32.618998 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:32.619000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:32.619377 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:32.619697 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:32.619698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:32.619889 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:32.620288 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:32.620295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:32.620423 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 14:30:32.620514 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:30:32.620692 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":96,"thread_start_us":63,"threads_started":1}
I20260812 14:30:32.620837 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.609649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:42713 (local address 127.23.173.194:39383)
0812 14:30:32.609781 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:32.609789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:32.609807 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:30:32.610272 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:32.610274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:32.610320 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:32.610452 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:32.611128 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.612168 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.613599 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.614211 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.614320 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:32.617253 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:32.617257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:32.617257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:32.617267 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:32.618908 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619477 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:32.619480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:32.619481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:32.619624 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619970 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:32.619971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:32.619971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:32.620212 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:32.620291 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:32.620585 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 14:30:32.620720 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260812 14:30:32.620891 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:32.609464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:46253 (local address 127.23.173.193:38055)
0812 14:30:32.609577 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:32.609589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:32.609645 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:32.610415 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:32.610418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:32.610834 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:32.611016 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:32.611021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.612309 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.612310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:32.613497 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:32.613501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:32.614332 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:32.614334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:32.614441 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:32.615114 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:32.615118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:32.617161 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:32.619068 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:32.619069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:32.619377 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:32.619697 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:32.619698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:32.619698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:32.619889 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:32.620286 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:32.620294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:32.620424 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 14:30:32.620513 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:30:32.620773 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":144}
W20260812 14:30:32.621670 17008 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
W20260812 14:30:32.621825 17003 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:32.621873 17003 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d; no voters: 3a1a820dbaf84193b40fdf3cada07cfe, 90d9d5f039e841c394cebd17bd4b6f80
I20260812 14:30:32.621966 17701 raft_consensus.cc:2726] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:30:33.430781 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:33.430539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:35071)
0812 14:30:33.430691 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:33.430730 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260812 14:30:33.471307 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:33.471090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:48125)
0812 14:30:33.471234 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:33.471270 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260812 14:30:33.600000 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:33.599804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:37011)
0812 14:30:33.599934 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:33.599963 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260812 14:30:34.431598 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:34.431309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:54103)
0812 14:30:34.431497 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:34.431548 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260812 14:30:34.472025 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:34.471807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:39167)
0812 14:30:34.471938 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:34.471980 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260812 14:30:34.600651 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:34.600440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:60241)
0812 14:30:34.600564 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:34.600605 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260812 14:30:34.802826 17715 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:34.802924 17715 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:34.803110 17715 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
W20260812 14:30:34.803536 17008 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
W20260812 14:30:34.803598 17003 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:34.803697 17008 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d; no voters: 3a1a820dbaf84193b40fdf3cada07cfe, 90d9d5f039e841c394cebd17bd4b6f80
I20260812 14:30:34.803805 17715 raft_consensus.cc:2726] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:30:35.432389 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:35.432153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:56007)
0812 14:30:35.432294 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:35.432337 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260812 14:30:35.472625 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:35.472410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:41997)
0812 14:30:35.472541 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:35.472581 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260812 14:30:35.601269 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:35.601053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:49331)
0812 14:30:35.601193 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:35.601227 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260812 14:30:36.433211 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:36.432944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:47369)
0812 14:30:36.433114 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:36.433161 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260812 14:30:36.473312 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:36.473077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:48757)
0812 14:30:36.473223 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:36.473267 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260812 14:30:36.602097 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:36.601826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:46281)
0812 14:30:36.601989 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:36.602038 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260812 14:30:36.993819 17727 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:36.993937 17727 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:36.994110 17727 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
W20260812 14:30:36.994454 17008 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
W20260812 14:30:36.994570 17003 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:36.994637 17003 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d; no voters: 3a1a820dbaf84193b40fdf3cada07cfe, 90d9d5f039e841c394cebd17bd4b6f80
I20260812 14:30:36.994709 17727 raft_consensus.cc:2726] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:30:37.433940 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:37.433683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:57517)
0812 14:30:37.433843 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:37.433886 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260812 14:30:37.473940 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:37.473733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:38721)
0812 14:30:37.473852 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:37.473889 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260812 14:30:37.602828 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:37.602558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:60969)
0812 14:30:37.602749 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:37.602781 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260812 14:30:38.434695 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:38.434467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:45173)
0812 14:30:38.434612 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:38.434645 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260812 14:30:38.474649 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:38.474455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:53445)
0812 14:30:38.474571 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:38.474608 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260812 14:30:38.603466 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:38.603254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:35941)
0812 14:30:38.603379 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:38.603417 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260812 14:30:39.435612 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:39.435301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:44209)
0812 14:30:39.435506 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:39.435560 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260812 14:30:39.475286 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:39.475058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:44181)
0812 14:30:39.475194 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:39.475240 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260812 14:30:39.604212 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:39.603968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:37701)
0812 14:30:39.604118 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:39.604163 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260812 14:30:40.436374 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:40.436147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:33273)
0812 14:30:40.436282 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:40.436324 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260812 14:30:40.475950 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:40.475741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:46813)
0812 14:30:40.475867 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:40.475902 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260812 14:30:40.604887 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:40.604655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:38909)
0812 14:30:40.604795 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:40.604835 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260812 14:30:41.437139 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:41.436912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:33215)
0812 14:30:41.437043 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:41.437087 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260812 14:30:41.476558 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:41.476346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:59273)
0812 14:30:41.476467 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:41.476504 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260812 14:30:41.605551 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:41.605312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:48161)
0812 14:30:41.605456 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:41.605504 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260812 14:30:41.617291 17756 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:41.617396 17756 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:41.617579 17756 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
W20260812 14:30:41.617967 17008 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
W20260812 14:30:41.618072 17003 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:41.618117 17003 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d; no voters: 3a1a820dbaf84193b40fdf3cada07cfe, 90d9d5f039e841c394cebd17bd4b6f80
I20260812 14:30:41.618187 17756 raft_consensus.cc:2726] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:30:42.437898 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:42.437632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:54079)
0812 14:30:42.437798 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:42.437839 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260812 14:30:42.477296 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:42.477045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:45833)
0812 14:30:42.477204 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:42.477250 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260812 14:30:42.606211 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:42.605979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:60775)
0812 14:30:42.606127 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:42.606165 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260812 14:30:43.438612 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:43.438376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:47845)
0812 14:30:43.438528 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:43.438562 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260812 14:30:43.477947 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:43.477754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:39209)
0812 14:30:43.477868 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:43.477904 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260812 14:30:43.606833 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:43.606628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:36259)
0812 14:30:43.606753 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:43.606790 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260812 14:30:44.138738 17765 raft_consensus.cc:482] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:44.138856 17765 raft_consensus.cc:504] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1a820dbaf84193b40fdf3cada07cfe" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39383 } } peers { permanent_uuid: "90d9d5f039e841c394cebd17bd4b6f80" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 46253 } } peers { permanent_uuid: "0cf5d3e39ec2447cadfa700ebbbfc84d" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 38443 } }
I20260812 14:30:44.139016 17765 leader_election.cc:288] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383), 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253)
W20260812 14:30:44.139438 17008 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a1a820dbaf84193b40fdf3cada07cfe (127.23.173.194:39383): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
W20260812 14:30:44.139544 17003 leader_election.cc:341] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d9d5f039e841c394cebd17bd4b6f80 (127.23.173.195:46253): Not found: Tablet not found: d4ad73836dec43a3aa189e4ec26f7f8b
I20260812 14:30:44.139590 17003 leader_election.cc:302] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cf5d3e39ec2447cadfa700ebbbfc84d; no voters: 3a1a820dbaf84193b40fdf3cada07cfe, 90d9d5f039e841c394cebd17bd4b6f80
I20260812 14:30:44.139683 17765 raft_consensus.cc:2726] T d4ad73836dec43a3aa189e4ec26f7f8b P 0cf5d3e39ec2447cadfa700ebbbfc84d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:30:44.439738 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:44.439330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:45265)
0812 14:30:44.439557 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:44.439628 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":155,"threads_started":1}
I20260812 14:30:44.478772 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:44.478438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:48745)
0812 14:30:44.478630 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:44.478692 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260812 14:30:44.607656 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:44.607334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:33029)
0812 14:30:44.607529 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:44.607583 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260812 14:30:45.440610 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:45.440345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:41857)
0812 14:30:45.440511 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:45.440558 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260812 14:30:45.479457 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:45.479245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:38991)
0812 14:30:45.479375 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:45.479414 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260812 14:30:45.608537 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:45.608253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:42883)
0812 14:30:45.608436 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:45.608486 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260812 14:30:46.441273 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:46.441061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.193:35885)
0812 14:30:46.441187 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:46.441225 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260812 14:30:46.480263 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:46.479998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.194:45383)
0812 14:30:46.480181 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:46.480217 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260812 14:30:46.608180 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 16925
I20260812 14:30:46.615302 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:46.613398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:38695 (local address 127.23.173.195:55941)
0812 14:30:46.615200 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:46.615252 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1781,"thread_start_us":1735,"threads_started":1}
I20260812 14:30:46.624388 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 17217
I20260812 14:30:46.641428 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 17467
2026-08-12T14:30:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18864 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:30:46 dist-test-slave-1l3l krb5kdc[17789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:30:46 dist-test-slave-1l3l krb5kdc[17789](info): set up 2 sockets
Aug 12 14:30:46 dist-test-slave-1l3l krb5kdc[17789](info): commencing operation
krb5kdc: starting...
W20260812 14:30:47.832216 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:30:47 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:30:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:30:47.954272 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41375
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:48.036158 17811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:48.036365 17811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:48.036393 17811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:48.037523 17811 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:30:48.037562 17811 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:30:48.037578 17811 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:30:48.037593 17811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:30:48.037611 17811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:48.037626 17811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:48.037643 17811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:30:48.037662 17811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:30:48.039145 17811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:48.039397 17811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:48.039621 17811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:48.044322 17823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:48.044405 17819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:48.044458 17811 server_base.cc:1034] running on GCE node
W20260812 14:30:48.044407 17820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:48.044821 17811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:48.045115 17811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:48.046262 17811 hybrid_clock.cc:648] HybridClock initialized: now 1786545048046241 us; error 32 us; skew 500 ppm
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:48.048695 17811 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:30:48.049146 17811 webserver.cc:460] Webserver started at http://127.23.173.254:33835/ using document root <none> and password file <none>
I20260812 14:30:48.049326 17811 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:48.049365 17811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:48.049458 17811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:48.050496 17811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "be698f77bec4420cab0540d3fcdd86aa"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20959e9242b42cb8ba8604d6bc6de0d7"
server_key_iv: "4541c1792eaa66fbc1c0dd5d6f0a2b5a"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.050803 17811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "be698f77bec4420cab0540d3fcdd86aa"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20959e9242b42cb8ba8604d6bc6de0d7"
server_key_iv: "4541c1792eaa66fbc1c0dd5d6f0a2b5a"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.052178 17811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:30:48.052841 17838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.053015 17811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:48.053094 17811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "be698f77bec4420cab0540d3fcdd86aa"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20959e9242b42cb8ba8604d6bc6de0d7"
server_key_iv: "4541c1792eaa66fbc1c0dd5d6f0a2b5a"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.053150 17811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:48.075047 17811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:48.077337 17811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:48.077471 17811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:48.081367 17811 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:41375
I20260812 14:30:48.081421 17922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:41375 every 8 connection(s)
I20260812 14:30:48.081753 17811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:30:48.082643 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:48.084954 17924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap starting.
I20260812 14:30:48.085530 17924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:48.085767 17924 log.cc:824] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:48.086376 17924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: No bootstrap required, opened a new log
I20260812 14:30:48.087908 17924 raft_consensus.cc:348] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:30:48.088044 17924 raft_consensus.cc:374] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:48.088068 17924 raft_consensus.cc:729] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be698f77bec4420cab0540d3fcdd86aa, State: Initialized, Role: FOLLOWER
I20260812 14:30:48.088181 17924 consensus_queue.cc:260] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:30:48.088256 17924 raft_consensus.cc:388] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:30:48.088284 17924 raft_consensus.cc:482] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:30:48.088317 17924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:48.088809 17924 raft_consensus.cc:504] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:30:48.088891 17924 leader_election.cc:302] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be698f77bec4420cab0540d3fcdd86aa; no voters:
I20260812 14:30:48.089025 17924 leader_election.cc:288] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:30:48.089071 17929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:48.089267 17924 sys_catalog.cc:564] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:30:48.089255 17929 raft_consensus.cc:686] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 LEADER]: Becoming Leader. State: Replica: be698f77bec4420cab0540d3fcdd86aa, State: Running, Role: LEADER
I20260812 14:30:48.089401 17929 consensus_queue.cc:237] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:30:48.089679 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 17811
I20260812 14:30:48.089798 17930 sys_catalog.cc:455] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be698f77bec4420cab0540d3fcdd86aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } } }
I20260812 14:30:48.089875 17930 sys_catalog.cc:458] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:48.090034 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:30:48.090054 17931 sys_catalog.cc:455] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader be698f77bec4420cab0540d3fcdd86aa. Latest consensus state: current_term: 1 leader_uuid: "be698f77bec4420cab0540d3fcdd86aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } } }
I20260812 14:30:48.090322 24247 external_mini_cluster.cc:1428] Setting key 0abfb4b8689e069290ac2efc9647cafd
I20260812 14:30:48.090348 17931 sys_catalog.cc:458] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: This master's current role is: LEADER
I20260812 14:30:48.090716 17936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:30:48.091375 17936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:30:48.093063 17936 catalog_manager.cc:1347] Generated new cluster ID: 82d000c90b654ff0bd46aa81caeea6d3
I20260812 14:30:48.093115 17936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:48.098007 17936 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:30:48.098558 17936 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:30:48.101432 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.090777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.23.173.254:41375)
0812 14:30:48.091004 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.091014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.091050 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:30:48.091157 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.091162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.091213 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.091358 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:48.092132 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.093326 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.094377 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.094986 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.095078 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.097931 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:48.097935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:48.097935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:48.097947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:48.099894 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.100271 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.100273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.100274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.100370 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.100559 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.100560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.100560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.100801 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:48.100936 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.101053 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.101193 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260812 14:30:48.105842 17936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Generated new TSK 0
I20260812 14:30:48.106007 17936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:30:48.134738 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41375
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:48.229691 17966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:48.229925 17966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:48.229959 17966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:48.231168 17966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:48.231218 17966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:48.231256 17966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:30:48.232825 17966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:48.233074 17966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:48.233301 17966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:48.238401 17981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:48.238402 17976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:48.238405 17977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:48.238672 17966 server_base.cc:1034] running on GCE node
I20260812 14:30:48.238867 17966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:48.239125 17966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:48.240273 17966 hybrid_clock.cc:648] HybridClock initialized: now 1786545048240259 us; error 29 us; skew 500 ppm
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:48.242851 17966 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:30:48.243309 17966 webserver.cc:460] Webserver started at http://127.23.173.193:34847/ using document root <none> and password file <none>
I20260812 14:30:48.243520 17966 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:48.243572 17966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:48.243733 17966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:48.244819 17966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "c53226e7ce9e4ab792894775a396a72a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20c6037a208c4e95614e854ac44415df"
server_key_iv: "22d89131b0e6308ffba65bd8d196ac5b"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.245173 17966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "c53226e7ce9e4ab792894775a396a72a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20c6037a208c4e95614e854ac44415df"
server_key_iv: "22d89131b0e6308ffba65bd8d196ac5b"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.246439 17966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:30:48.247138 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.247289 17966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:48.247375 17966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "c53226e7ce9e4ab792894775a396a72a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20c6037a208c4e95614e854ac44415df"
server_key_iv: "22d89131b0e6308ffba65bd8d196ac5b"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.247438 17966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:48.268666 17966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:48.270925 17966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:48.271056 17966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:48.271294 17966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:48.271626 17966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:48.271662 17966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.271692 17966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:48.271731 17966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.277459 17966 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:43287
I20260812 14:30:48.277536 18174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:43287 every 8 connection(s)
I20260812 14:30:48.277837 17966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:30:48.280920 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 17966
I20260812 14:30:48.281035 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:30:48.281229 24247 external_mini_cluster.cc:1428] Setting key 0aec29500aa664bf4b64af60ee6e3ff5
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:48.287096 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.278539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:56185 (local address 127.23.173.254:41375)
0812 14:30:48.278602 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.278605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.278791 (+ 186us) server_negotiation.cc:407] Connection header received
0812 14:30:48.279316 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.279317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.279335 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.279390 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:48.280375 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.280992 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.282164 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.282413 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.282486 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.284532 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:48.284534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:48.284535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:48.284551 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:48.285829 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.286202 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.286204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.286204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.286259 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.286479 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.286480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.286480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.286597 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:48.286655 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.286913 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.287006 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:30:48.287256 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.278509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:56185)
0812 14:30:48.278669 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.278706 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.279211 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.279482 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.279486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.280079 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:48.280274 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.280279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.281101 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.281102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.282083 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.282086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.282486 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.282487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.282583 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.283093 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:48.283101 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:48.284442 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:48.285906 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:48.285914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:48.285916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:48.286135 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:48.286314 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:48.286314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:48.286315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:48.286422 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:48.286656 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:48.286663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:48.286796 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.286884 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.287046 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260812 14:30:48.287729 18175 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41375
I20260812 14:30:48.287820 18175 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:48.287994 18175 heartbeater.cc:511] Master 127.23.173.254:41375 requested a full tablet report, sending...
I20260812 14:30:48.288496 17873 ts_manager.cc:194] Registered new tserver with Master: c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287)
I20260812 14:30:48.289140 17873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:56185
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:30:48.306855 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41375
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:48.389986 18182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:48.390244 18182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:48.390290 18182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:48.391551 18182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:48.391599 18182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:48.391628 18182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:30:48.393244 18182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:48.393489 18182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:48.393724 18182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:30:48.398962 18182 server_base.cc:1034] running on GCE node
W20260812 14:30:48.398903 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:48.398903 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:48.398931 18190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:48.399379 18182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:48.399645 18182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:48.400789 18182 hybrid_clock.cc:648] HybridClock initialized: now 1786545048400784 us; error 41 us; skew 500 ppm
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:48.403332 18182 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:30:48.403849 18182 webserver.cc:460] Webserver started at http://127.23.173.194:45887/ using document root <none> and password file <none>
I20260812 14:30:48.404083 18182 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:48.404135 18182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:48.404268 18182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:48.405335 18182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "b45df406c7a442cd80b31f7b68f34c3a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "0feebdce9e6647b25d0511116644c144"
server_key_iv: "b433af20c977aabd852d7f602a52dec4"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.405696 18182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "b45df406c7a442cd80b31f7b68f34c3a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "0feebdce9e6647b25d0511116644c144"
server_key_iv: "b433af20c977aabd852d7f602a52dec4"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.407079 18182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:30:48.407801 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.407951 18182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:48.408033 18182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "b45df406c7a442cd80b31f7b68f34c3a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "0feebdce9e6647b25d0511116644c144"
server_key_iv: "b433af20c977aabd852d7f602a52dec4"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.408114 18182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:48.419406 18182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:48.421691 18182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:48.421818 18182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:48.422014 18182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:48.422348 18182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:48.422396 18182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.422418 18182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:48.422429 18182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.428577 18182 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:39479
I20260812 14:30:48.428642 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:39479 every 8 connection(s)
I20260812 14:30:48.428962 18182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:30:48.432508 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 18182
I20260812 14:30:48.432610 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:30:48.432796 24247 external_mini_cluster.cc:1428] Setting key 25c497e4b44c6d98772f3b3b4c6eeb6e
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:48.439419 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.429723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:33299 (local address 127.23.173.254:41375)
0812 14:30:48.429791 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.429793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.429994 (+ 201us) server_negotiation.cc:407] Connection header received
0812 14:30:48.430530 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.430532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.430544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.430600 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:48.431690 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.432225 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.433392 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.433650 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.433725 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.436298 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:48.436299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:48.436299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:48.436307 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:48.437786 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.438232 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.438234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.438234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.438313 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.438674 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.438674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.438675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.438836 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:48.438912 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.439215 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.439316 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:30:48.439546 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.429678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:33299)
0812 14:30:48.429862 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.429902 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.430415 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.430691 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.430695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.431312 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:48.431580 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.431585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.432327 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.432328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.433273 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.433276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.433747 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.433748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.433887 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.434404 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:48.434412 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:48.436179 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:48.437895 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:48.437904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:48.437907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:48.438146 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:48.438436 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:48.438437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:48.438437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:48.438581 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:48.438929 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:48.438936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:48.439075 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.439176 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.439320 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260812 14:30:48.440189 18415 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41375
I20260812 14:30:48.440299 18415 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:48.440529 18415 heartbeater.cc:511] Master 127.23.173.254:41375 requested a full tablet report, sending...
I20260812 14:30:48.440948 17873 ts_manager.cc:194] Registered new tserver with Master: b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479)
I20260812 14:30:48.441474 17873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:33299
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:30:48.463697 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41375
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:30:48.547355 18426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:30:48.547583 18426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:30:48.547607 18426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:30:48.548806 18426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:30:48.548861 18426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:30:48.548887 18426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:30:48.550360 18426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:30:48.550597 18426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:30:48.550812 18426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:30:48.555874 18435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:30:48.555897 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:48.556049 18426 server_base.cc:1034] running on GCE node
W20260812 14:30:48.556137 18438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:30:48.556367 18426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:30:48.556644 18426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:30:48.557801 18426 hybrid_clock.cc:648] HybridClock initialized: now 1786545048557787 us; error 29 us; skew 500 ppm
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:30:48.560315 18426 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:30:48.560774 18426 webserver.cc:460] Webserver started at http://127.23.173.195:41939/ using document root <none> and password file <none>
I20260812 14:30:48.560987 18426 fs_manager.cc:359] Metadata directory not provided
I20260812 14:30:48.561029 18426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:30:48.561124 18426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:30:48.562125 18426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "685480c9ec4fd341a95eaca468a6d831"
server_key_iv: "ac8eeb9fdfd02fca397baa4265c39b72"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.562429 18426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "685480c9ec4fd341a95eaca468a6d831"
server_key_iv: "ac8eeb9fdfd02fca397baa4265c39b72"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.563684 18426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:30:48.564401 18453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.564576 18426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:30:48.564661 18426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "685480c9ec4fd341a95eaca468a6d831"
server_key_iv: "ac8eeb9fdfd02fca397baa4265c39b72"
server_key_version: "encryptionkey@0"
I20260812 14:30:48.564718 18426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:30:48.579269 18426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:30:48.581415 18426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:30:48.581542 18426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:30:48.581746 18426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:30:48.582006 18426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:30:48.582036 18426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.582057 18426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:30:48.582072 18426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:30:48.587584 18426 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:35807
I20260812 14:30:48.587656 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:35807 every 8 connection(s)
I20260812 14:30:48.587968 18426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:30:48.589812 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 18426
I20260812 14:30:48.589907 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:30:48.590094 24247 external_mini_cluster.cc:1428] Setting key 427eaae3c665f96b8374868e428cf21b
Aug 12 14:30:48 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:30:48.596951 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.588667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:37303 (local address 127.23.173.254:41375)
0812 14:30:48.588714 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.588716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.588943 (+ 227us) server_negotiation.cc:407] Connection header received
0812 14:30:48.589485 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.589486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.589499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.589553 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:48.590735 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.591340 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.592375 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.592562 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.592606 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.594583 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:48.594584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:48.594584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:48.594590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:48.595894 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.596248 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.596249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.596249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.596297 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.596479 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.596480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.596480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.596573 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:48.596615 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.596822 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.596882 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:30:48.597118 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.588638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:37303)
0812 14:30:48.588793 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.588842 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.589365 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.589650 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.589654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.590363 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:30:48.590657 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.590664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.591422 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.591423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.592302 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.592305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.592621 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.592622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.592724 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.593143 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:30:48.593151 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:30:48.594495 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:30:48.595972 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:48.595979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:48.595981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:48.596193 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:48.596342 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:30:48.596343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:30:48.596343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:30:48.596433 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:30:48.596613 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:30:48.596616 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:30:48.596727 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.596803 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.596922 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260812 14:30:48.597538 18657 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41375
I20260812 14:30:48.597626 18657 heartbeater.cc:464] Registering TS with master...
I20260812 14:30:48.597776 18657 heartbeater.cc:511] Master 127.23.173.254:41375 requested a full tablet report, sending...
I20260812 14:30:48.598120 17873 ts_manager.cc:194] Registered new tserver with Master: cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
I20260812 14:30:48.598471 17873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:37303
I20260812 14:30:48.601764 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:30:48.608312 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.602998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.23.173.254:41375)
0812 14:30:48.603045 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.603047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.603144 (+ 97us) server_negotiation.cc:407] Connection header received
0812 14:30:48.603207 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.603208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.603217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.603264 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:30:48.603803 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.604284 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.604976 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.605156 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.605210 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.606271 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:30:48.606273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:30:48.606273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:30:48.606280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:30:48.607289 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.607588 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.607589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.607589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.607638 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:30:48.607821 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:30:48.607822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:30:48.607822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:30:48.607956 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:30:48.608041 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.608156 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.608228 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:30:48.609656 17873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:30:48.611200 17873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:30:48.617079 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.613338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43287 (local address 127.0.0.1:36432)
0812 14:30:48.613488 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.613531 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.613617 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.613938 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.613941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.613943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:48.614017 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.614019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.615028 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.615029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.616028 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.616031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.616791 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.616792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.616868 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.616881 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.616950 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.617006 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260812 14:30:48.617090 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.613459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.23.173.193:43287)
0812 14:30:48.613773 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.613777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.613786 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:48.613827 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.613829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.613844 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.613929 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:48.614092 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.614930 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.616108 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.616716 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.616780 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.616835 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.616966 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.617004 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":93,"threads_started":1}
I20260812 14:30:48.617491 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.613339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39479 (local address 127.0.0.1:34278)
0812 14:30:48.613666 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.613677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.613735 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.613944 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.613945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.613947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:48.614015 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.614017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.615010 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.615011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.616290 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.616294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.617262 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.617263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.617333 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.617336 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.617385 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.617421 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":27,"thread_start_us":113,"threads_started":1}
I20260812 14:30:48.617529 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.613467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.23.173.194:39479)
0812 14:30:48.613690 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.613696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.613797 (+ 101us) server_negotiation.cc:407] Connection header received
0812 14:30:48.613842 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.613845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.613860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.613930 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:48.614084 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.614924 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.616395 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.617116 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.617183 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.617229 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.617409 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.617457 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260812 14:30:48.617942 18072 tablet_service.cc:1467] Processing CreateTablet for tablet aad4a1b13c684c98b913661ca7890576 (DEFAULT_TABLE table=test-table [id=70951dfabbe34646ab61dba056058ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:48.618243 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad4a1b13c684c98b913661ca7890576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:48.618486 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.613559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35807 (local address 127.0.0.1:48478)
0812 14:30:48.613762 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.613770 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.613811 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.615261 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.615261 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.615264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:48.615308 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.615310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.616568 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.616569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.617601 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.617604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.618275 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.618276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.618336 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.618340 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.618383 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.618422 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":69,"threads_started":1}
I20260812 14:30:48.618516 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.613757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.23.173.195:35807)
0812 14:30:48.615106 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.615110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.615119 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:48.615155 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.615157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.615173 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.615249 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:48.615393 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.616471 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.617688 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.618200 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.618267 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.618318 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.618406 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.618444 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":85,"threads_started":1}
I20260812 14:30:48.619249 18544 tablet_service.cc:1467] Processing CreateTablet for tablet aad4a1b13c684c98b913661ca7890576 (DEFAULT_TABLE table=test-table [id=70951dfabbe34646ab61dba056058ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:48.619529 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad4a1b13c684c98b913661ca7890576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:48.620357 18307 tablet_service.cc:1467] Processing CreateTablet for tablet aad4a1b13c684c98b913661ca7890576 (DEFAULT_TABLE table=test-table [id=70951dfabbe34646ab61dba056058ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:30:48.620683 18307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad4a1b13c684c98b913661ca7890576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:30:48.621654 18687 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Bootstrap starting.
I20260812 14:30:48.622403 18687 tablet_bootstrap.cc:654] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:48.622605 18688 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Bootstrap starting.
I20260812 14:30:48.622967 18687 log.cc:824] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:48.623252 18688 tablet_bootstrap.cc:654] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:48.623659 18690 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Bootstrap starting.
I20260812 14:30:48.623726 18687 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: No bootstrap required, opened a new log
I20260812 14:30:48.623785 18687 ts_tablet_manager.cc:1397] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:48.623886 18688 log.cc:824] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:48.624424 18690 tablet_bootstrap.cc:654] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Neither blocks nor log segments found. Creating new log.
I20260812 14:30:48.624624 18688 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: No bootstrap required, opened a new log
I20260812 14:30:48.624686 18688 ts_tablet_manager.cc:1397] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:30:48.625062 18690 log.cc:824] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:30:48.625895 18690 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: No bootstrap required, opened a new log
I20260812 14:30:48.625854 18687 raft_consensus.cc:348] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.626015 18690 ts_tablet_manager.cc:1397] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:30:48.626034 18687 raft_consensus.cc:374] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:48.626071 18687 raft_consensus.cc:729] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53226e7ce9e4ab792894775a396a72a, State: Initialized, Role: FOLLOWER
I20260812 14:30:48.626220 18687 consensus_queue.cc:260] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.626507 18687 ts_tablet_manager.cc:1428] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:30:48.626441 18688 raft_consensus.cc:348] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.626577 18688 raft_consensus.cc:374] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:48.626641 18175 heartbeater.cc:503] Master 127.23.173.254:41375 was elected leader, sending a full tablet report...
I20260812 14:30:48.626649 18688 raft_consensus.cc:729] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd04a39700374b35a094d2c5a4ab1f0a, State: Initialized, Role: FOLLOWER
I20260812 14:30:48.626763 18688 consensus_queue.cc:260] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.627022 18688 ts_tablet_manager.cc:1428] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:30:48.627128 18657 heartbeater.cc:503] Master 127.23.173.254:41375 was elected leader, sending a full tablet report...
I20260812 14:30:48.628262 18690 raft_consensus.cc:348] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.628387 18690 raft_consensus.cc:374] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:30:48.628409 18690 raft_consensus.cc:729] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45df406c7a442cd80b31f7b68f34c3a, State: Initialized, Role: FOLLOWER
I20260812 14:30:48.628512 18690 consensus_queue.cc:260] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.628758 18690 ts_tablet_manager.cc:1428] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:30:48.628803 18415 heartbeater.cc:503] Master 127.23.173.254:41375 was elected leader, sending a full tablet report...
I20260812 14:30:48.667821 18701 raft_consensus.cc:482] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:30:48.667950 18701 raft_consensus.cc:504] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.668354 18701 leader_election.cc:288] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
I20260812 14:30:48.672250 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.668721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:55231 (local address 127.23.173.195:35807)
0812 14:30:48.668766 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.668768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.668775 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:30:48.668803 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.668804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.668812 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.668873 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:48.669036 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.669795 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.671291 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.671894 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.671970 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.671979 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.672139 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.672177 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:30:48.672250 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.668603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:36103 (local address 127.23.173.193:43287)
0812 14:30:48.668641 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.668643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.668769 (+ 126us) server_negotiation.cc:407] Connection header received
0812 14:30:48.668816 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.668817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.668825 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.668877 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:30:48.669051 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.669806 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.671290 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.671907 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.671970 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.671979 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.672139 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.672178 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:30:48.672276 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.668510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35807 (local address 127.23.173.194:55231)
0812 14:30:48.668577 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.668591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.668640 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.668881 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.668882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.668884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:48.668971 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.668973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.669879 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.669880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.671202 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.671205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.671970 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.671971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.672071 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.672073 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.672128 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.672188 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":58}
I20260812 14:30:48.672281 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.668515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43287 (local address 127.23.173.194:36103)
0812 14:30:48.668714 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.668723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:30:48.668772 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:30:48.668890 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:30:48.668890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:30:48.668892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:30:48.668988 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.668990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.669879 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.669880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.671202 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:30:48.671205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.671971 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.671972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:30:48.672067 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.672071 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:30:48.672128 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:30:48.672186 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":139,"thread_start_us":130,"threads_started":1}
I20260812 14:30:48.672712 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4a1b13c684c98b913661ca7890576" candidate_uuid: "b45df406c7a442cd80b31f7b68f34c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c53226e7ce9e4ab792894775a396a72a" is_pre_election: true
I20260812 14:30:48.672807 18579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4a1b13c684c98b913661ca7890576" candidate_uuid: "b45df406c7a442cd80b31f7b68f34c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" is_pre_election: true
I20260812 14:30:48.672901 18100 raft_consensus.cc:2454] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45df406c7a442cd80b31f7b68f34c3a in term 0.
I20260812 14:30:48.672932 18579 raft_consensus.cc:2454] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45df406c7a442cd80b31f7b68f34c3a in term 0.
I20260812 14:30:48.673156 18210 leader_election.cc:302] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45df406c7a442cd80b31f7b68f34c3a, c53226e7ce9e4ab792894775a396a72a; no voters:
I20260812 14:30:48.673292 18701 raft_consensus.cc:2781] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:30:48.673352 18701 raft_consensus.cc:482] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:30:48.673373 18701 raft_consensus.cc:3037] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:48.674070 18701 raft_consensus.cc:504] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.674187 18701 leader_election.cc:288] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [CANDIDATE]: Term 1 election: Requested vote from peers c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
I20260812 14:30:48.674320 18579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4a1b13c684c98b913661ca7890576" candidate_uuid: "b45df406c7a442cd80b31f7b68f34c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
I20260812 14:30:48.674412 18579 raft_consensus.cc:3037] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:48.674381 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4a1b13c684c98b913661ca7890576" candidate_uuid: "b45df406c7a442cd80b31f7b68f34c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c53226e7ce9e4ab792894775a396a72a"
I20260812 14:30:48.674484 18100 raft_consensus.cc:3037] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:30:48.675381 18579 raft_consensus.cc:2454] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45df406c7a442cd80b31f7b68f34c3a in term 1.
I20260812 14:30:48.675381 18100 raft_consensus.cc:2454] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45df406c7a442cd80b31f7b68f34c3a in term 1.
I20260812 14:30:48.675554 18211 leader_election.cc:302] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45df406c7a442cd80b31f7b68f34c3a, cd04a39700374b35a094d2c5a4ab1f0a; no voters:
I20260812 14:30:48.675659 18701 raft_consensus.cc:2781] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:30:48.675832 18701 raft_consensus.cc:686] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 1 LEADER]: Becoming Leader. State: Replica: b45df406c7a442cd80b31f7b68f34c3a, State: Running, Role: LEADER
I20260812 14:30:48.675915 18701 consensus_queue.cc:237] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:30:48.676697 17873 catalog_manager.cc:5495] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a reported cstate change: term changed from 0 to 1, leader changed from <none> to b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194). New cstate: current_term: 1 leader_uuid: "b45df406c7a442cd80b31f7b68f34c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } health_report { overall_health: UNKNOWN } } }
W20260812 14:30:48.679476 18416 tablet.cc:2369] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:30:48.689173 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.687113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.23.173.194:39479)
0812 14:30:48.687157 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:30:48.687158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:30:48.687167 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:30:48.687221 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:30:48.687222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:30:48.687240 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:30:48.687295 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:30:48.687380 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.687865 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.688575 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:30:48.688749 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:30:48.688797 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:30:48.688918 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:30:48.689006 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:30:48.689073 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:30:48.689103 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 14:30:48.689120 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:30:48.690761 18579 raft_consensus.cc:1261] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Refusing update from remote peer b45df406c7a442cd80b31f7b68f34c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:48.690761 18100 raft_consensus.cc:1261] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Refusing update from remote peer b45df406c7a442cd80b31f7b68f34c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:30:48.691075 18710 consensus_queue.cc:1035] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:48.691215 18701 consensus_queue.cc:1035] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:30:48.693758 18717 mvcc.cc:205] Tried to move back new op lower bound from 7317688519435145216 to 7317688519377260544. Current Snapshot: MvccSnapshot[applied={T|T < 7317688519435145216}]
I20260812 14:30:48.694252 18715 mvcc.cc:205] Tried to move back new op lower bound from 7317688519435145216 to 7317688519377260544. Current Snapshot: MvccSnapshot[applied={T|T < 7317688519435145216 or (T in {7317688519435145216})}]
I20260812 14:30:48.694619 18714 mvcc.cc:205] Tried to move back new op lower bound from 7317688519435145216 to 7317688519377260544. Current Snapshot: MvccSnapshot[applied={T|T < 7317688519435145216 or (T in {7317688519435145216})}]
I20260812 14:30:48.696311 17873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:48.696398 17873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:30:48.697101 17873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Sending DeleteTablet for 3 replicas of tablet aad4a1b13c684c98b913661ca7890576
I20260812 14:30:48.697461 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 17966
I20260812 14:30:48.697443 18072 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4a1b13c684c98b913661ca7890576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:48 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36432
I20260812 14:30:48.697491 18544 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4a1b13c684c98b913661ca7890576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:48 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:48478
I20260812 14:30:48.700213 18307 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4a1b13c684c98b913661ca7890576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:48 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:34278
I20260812 14:30:48.722108 18729 tablet_replica.cc:330] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: stopping tablet replica
I20260812 14:30:48.722326 18729 raft_consensus.cc:2229] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:30:48.722375 18727 tablet_replica.cc:330] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: stopping tablet replica
I20260812 14:30:48.722543 18729 raft_consensus.cc:2258] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:48.722612 18727 raft_consensus.cc:2229] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:30:48.722967 18727 raft_consensus.cc:2258] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:30:48.723121 18729 ts_tablet_manager.cc:1905] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:48.723520 18727 ts_tablet_manager.cc:1905] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:30:48.725281 18727 ts_tablet_manager.cc:1918] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:48.725294 18729 ts_tablet_manager.cc:1918] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:30:48.725366 18729 log.cc:1196] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/aad4a1b13c684c98b913661ca7890576
I20260812 14:30:48.725436 18727 log.cc:1196] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/aad4a1b13c684c98b913661ca7890576
I20260812 14:30:48.725735 18729 ts_tablet_manager.cc:1939] T aad4a1b13c684c98b913661ca7890576 P b45df406c7a442cd80b31f7b68f34c3a: Deleting consensus metadata
I20260812 14:30:48.725814 18727 ts_tablet_manager.cc:1939] T aad4a1b13c684c98b913661ca7890576 P cd04a39700374b35a094d2c5a4ab1f0a: Deleting consensus metadata
I20260812 14:30:48.726245 17845 catalog_manager.cc:4841] TS b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): tablet aad4a1b13c684c98b913661ca7890576 (table test-table [id=70951dfabbe34646ab61dba056058ebc]) successfully deleted
I20260812 14:30:48.726359 17845 catalog_manager.cc:4841] TS cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): tablet aad4a1b13c684c98b913661ca7890576 (table test-table [id=70951dfabbe34646ab61dba056058ebc]) successfully deleted
W20260812 14:30:48.727418 17843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:43287 (error 108)
I20260812 14:30:48.727651 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 18182
I20260812 14:30:48.745635 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:30:48.745462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43287 (local address 127.0.0.1:36444)
0812 14:30:48.745522 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:30:48.745585 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 14:30:48.745811 17843 catalog_manager.cc:4584] TS c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aad4a1b13c684c98b913661ca7890576: Network error: Client connection negotiation failed: client connection to 127.23.173.193:43287: connect: Connection refused (error 111)
I20260812 14:30:48.749084 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 18426
I20260812 14:30:48.759554 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 17811
I20260812 14:31:08.767614 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41375
--webserver_interface=127.23.173.254
--webserver_port=33835
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:08.849586 18733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:08.849784 18733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:08.849805 18733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:08.850883 18733 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:31:08.850919 18733 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:31:08.850931 18733 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:31:08.850941 18733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:08.850958 18733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:08.850970 18733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:08.850986 18733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:08.850998 18733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:08.852473 18733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:08.852677 18733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:08.852895 18733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:08.862073 18747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:08.862097 18745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:08.862087 18744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:08.862085 18733 server_base.cc:1034] running on GCE node
I20260812 14:31:08.862529 18733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:08.862833 18733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:08.863987 18733 hybrid_clock.cc:648] HybridClock initialized: now 1786545068863972 us; error 33 us; skew 500 ppm
Aug 12 14:31:08 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:08.872359 18733 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:31:08.872879 18733 webserver.cc:460] Webserver started at http://127.23.173.254:33835/ using document root <none> and password file <none>
I20260812 14:31:08.873124 18733 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:08.873188 18733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:08.874571 18733 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:08.875267 18761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:08.875444 18733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:08.875540 18733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "be698f77bec4420cab0540d3fcdd86aa"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20959e9242b42cb8ba8604d6bc6de0d7"
server_key_iv: "4541c1792eaa66fbc1c0dd5d6f0a2b5a"
server_key_version: "encryptionkey@0"
I20260812 14:31:08.876014 18733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:08.894673 18733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:08.896663 18733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:08.896795 18733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:08.900831 18733 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:41375
I20260812 14:31:08.900868 18847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:41375 every 8 connection(s)
I20260812 14:31:08.901240 18733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:08.902750 18849 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:31:08.903434 18849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap starting.
I20260812 14:31:08.903458 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 18733
I20260812 14:31:08.903756 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:43287
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:41375
--webserver_port=34847
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:31:08.911648 18849 log.cc:824] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:08.913769 18849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:31:08.914064 18849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap complete.
I20260812 14:31:08.916395 18849 raft_consensus.cc:348] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:08.916625 18849 raft_consensus.cc:729] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be698f77bec4420cab0540d3fcdd86aa, State: Initialized, Role: FOLLOWER
I20260812 14:31:08.916774 18849 consensus_queue.cc:260] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:08.916847 18849 raft_consensus.cc:388] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:08.916911 18849 raft_consensus.cc:482] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:08.916960 18849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:31:08.918042 18849 raft_consensus.cc:504] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:08.918140 18849 leader_election.cc:302] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be698f77bec4420cab0540d3fcdd86aa; no voters:
I20260812 14:31:08.918301 18849 leader_election.cc:288] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:31:08.918349 18855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:31:08.918526 18855 raft_consensus.cc:686] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 LEADER]: Becoming Leader. State: Replica: be698f77bec4420cab0540d3fcdd86aa, State: Running, Role: LEADER
I20260812 14:31:08.918574 18849 sys_catalog.cc:564] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:08.918618 18855 consensus_queue.cc:237] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:08.918906 18856 sys_catalog.cc:455] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be698f77bec4420cab0540d3fcdd86aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } } }
I20260812 14:31:08.919082 18856 sys_catalog.cc:458] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:08.919027 18855 sys_catalog.cc:455] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader be698f77bec4420cab0540d3fcdd86aa. Latest consensus state: current_term: 2 leader_uuid: "be698f77bec4420cab0540d3fcdd86aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } } }
I20260812 14:31:08.919284 18855 sys_catalog.cc:458] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:08.919517 18869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:08.920315 18869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:08.920428 18869 catalog_manager.cc:1259] Loaded cluster ID: 82d000c90b654ff0bd46aa81caeea6d3
I20260812 14:31:08.920460 18869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:08.921568 18869 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:08.921756 18869 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Loaded TSK: 0
I20260812 14:31:08.933298 18869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Generated new TSK 1
I20260812 14:31:08.933401 18869 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:31:08.986344 18853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:08.986550 18853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:08.986582 18853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:08.987751 18853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:08.987793 18853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:08.987829 18853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:08.989287 18853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:43287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=34847
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:08.989516 18853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:08.989727 18853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:08.994614 18894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:08.994626 18898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:08.994705 18893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:08.994984 18853 server_base.cc:1034] running on GCE node
I20260812 14:31:08.995182 18853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:08.995447 18853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:08.996593 18853 hybrid_clock.cc:648] HybridClock initialized: now 1786545068996577 us; error 33 us; skew 500 ppm
Aug 12 14:31:08 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:08.999264 18853 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:31:08.999749 18853 webserver.cc:460] Webserver started at http://127.23.173.193:34847/ using document root <none> and password file <none>
I20260812 14:31:08.999989 18853 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:09.000054 18853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:09.001355 18853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:09.001987 18912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.002225 18853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:09.002318 18853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "c53226e7ce9e4ab792894775a396a72a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20c6037a208c4e95614e854ac44415df"
server_key_iv: "22d89131b0e6308ffba65bd8d196ac5b"
server_key_version: "encryptionkey@0"
I20260812 14:31:09.002767 18853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:09.015764 18853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:09.017814 18853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:09.017961 18853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:09.018167 18853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:09.018666 18928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:31:09.020674 18853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:31:09.020730 18853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:31:09.020776 18853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:31:09.021427 18853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:31:09.021464 18853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:31:09.021549 18928 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Bootstrap starting.
I20260812 14:31:09.028149 18853 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:43287
I20260812 14:31:09.028594 18853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:31:09.029196 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 18853
I20260812 14:31:09.029496 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:39479
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:41375
--webserver_port=45887
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:31:09.032742 18928 log.cc:824] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:09.036368 18928 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:31:09.036628 18928 tablet_bootstrap.cc:492] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Bootstrap complete.
I20260812 14:31:09.037457 18928 ts_tablet_manager.cc:1397] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260812 14:31:09.038657 18928 raft_consensus.cc:348] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:31:09.038902 18928 raft_consensus.cc:729] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53226e7ce9e4ab792894775a396a72a, State: Initialized, Role: FOLLOWER
I20260812 14:31:09.039105 18928 consensus_queue.cc:260] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } }
I20260812 14:31:09.039392 18928 ts_tablet_manager.cc:1428] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260812 14:31:09.040305 19102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:43287 every 8 connection(s)
Aug 12 14:31:09 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:09.054564 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.044566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:35295 (local address 127.23.173.254:41375)
0812 14:31:09.045637 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.045645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.045677 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:31:09.045783 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.045788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.045829 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.045970 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:09.046645 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.047623 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.048638 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.049055 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.049153 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.051272 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:09.051276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:09.051276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:09.051287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:09.052950 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.053336 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.053340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.053340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.053438 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.053674 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.053675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.053676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.053892 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:09.053968 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.054246 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.054366 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":93,"threads_started":1}
I20260812 14:31:09.054576 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.044468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:35295)
0812 14:31:09.044657 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.044705 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.044835 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.046015 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.046020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.046296 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:09.046524 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.046529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.047753 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.047754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.048550 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.048553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.049141 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.049142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.049263 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.049780 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:09.049789 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:09.051150 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:09.053033 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:09.053041 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:09.053043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:09.053258 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:09.053507 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:09.053508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:09.053508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:09.053615 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:09.053961 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:09.053968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:09.054096 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.054181 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.054358 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260812 14:31:09.055538 19104 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41375
I20260812 14:31:09.055648 19104 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:09.055857 19104 heartbeater.cc:511] Master 127.23.173.254:41375 requested a full tablet report, sending...
I20260812 14:31:09.056564 18792 ts_manager.cc:194] Registered new tserver with Master: c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287)
I20260812 14:31:09.057605 18792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:35295
I20260812 14:31:09.058475 19104 heartbeater.cc:503] Master 127.23.173.254:41375 was elected leader, sending a full tablet report...
I20260812 14:31:09.062768 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.057384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.23.173.193:43287)
0812 14:31:09.060021 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.060025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.060031 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:31:09.060057 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.060058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.060069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.060131 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:09.060306 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.060950 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.061946 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.062389 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.062461 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.062514 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.062657 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.062695 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2610,"thread_start_us":71,"threads_started":1}
I20260812 14:31:09.062773 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.057132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43287 (local address 127.0.0.1:36086)
0812 14:31:09.059830 (+ 2698us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.059862 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.059919 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.060153 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.060156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.060158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:09.060222 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.060224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.061027 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.061028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.061851 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.061854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.062475 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.062476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.062560 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.062573 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.062635 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.062686 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"thread_start_us":102,"threads_started":1}
I20260812 14:31:09.063333 18999 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4a1b13c684c98b913661ca7890576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:30:48 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36086
I20260812 14:31:09.063910 19116 tablet_replica.cc:330] stopping tablet replica
I20260812 14:31:09.064011 19116 raft_consensus.cc:2229] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:09.064081 19116 raft_consensus.cc:2258] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:09.064375 19116 ts_tablet_manager.cc:1905] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:09.066413 19116 ts_tablet_manager.cc:1918] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:09.066481 19116 log.cc:1196] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal/wals/aad4a1b13c684c98b913661ca7890576
I20260812 14:31:09.066699 19116 ts_tablet_manager.cc:1939] T aad4a1b13c684c98b913661ca7890576 P c53226e7ce9e4ab792894775a396a72a: Deleting consensus metadata
I20260812 14:31:09.067176 18764 catalog_manager.cc:4841] TS c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287): tablet aad4a1b13c684c98b913661ca7890576 (table test-table [id=70951dfabbe34646ab61dba056058ebc]) successfully deleted
W20260812 14:31:09.123193 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:09.123386 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:09.123409 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:09.124576 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:09.124617 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:09.124641 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:09.126049 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:39479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=45887
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:09.126253 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:09.126463 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:09.131178 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:09.131244 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:09.131173 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:09.131439 19106 server_base.cc:1034] running on GCE node
I20260812 14:31:09.131628 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:09.131858 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:09.132995 19106 hybrid_clock.cc:648] HybridClock initialized: now 1786545069132986 us; error 29 us; skew 500 ppm
Aug 12 14:31:09 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545069, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:09.135427 19106 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:31:09.135883 19106 webserver.cc:460] Webserver started at http://127.23.173.194:45887/ using document root <none> and password file <none>
I20260812 14:31:09.136092 19106 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:09.136157 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:09.137488 19106 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:09.138059 19143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.138213 19106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:09.138294 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "b45df406c7a442cd80b31f7b68f34c3a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "0feebdce9e6647b25d0511116644c144"
server_key_iv: "b433af20c977aabd852d7f602a52dec4"
server_key_version: "encryptionkey@0"
I20260812 14:31:09.138612 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:09.151335 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:09.153067 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:09.153216 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:09.153431 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:09.153712 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:09.153745 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.153774 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:09.153791 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.159269 19106 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:39479
I20260812 14:31:09.159329 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:39479 every 8 connection(s)
I20260812 14:31:09.159637 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:31:09 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545069, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:09.167675 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 19106
I20260812 14:31:09.167944 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:35807
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:41375
--webserver_port=41939
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:31:09.168427 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.160322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:43487 (local address 127.23.173.254:41375)
0812 14:31:09.160396 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.160398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.160631 (+ 233us) server_negotiation.cc:407] Connection header received
0812 14:31:09.161103 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.161104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.161117 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.161160 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:09.162082 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.162592 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.163493 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.163823 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.163872 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.165895 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:09.165896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:09.165897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:09.165903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:09.167001 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.167397 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.167398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.167398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.167483 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.167734 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.167735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.167735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.167862 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:09.167920 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.168237 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.168327 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:31:09.168561 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.160312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:43487)
0812 14:31:09.160496 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.160536 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.161017 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.161245 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.161248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.161803 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:09.161995 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.162000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.162661 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.162662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.163428 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.163430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.163892 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.163893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.163999 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.164446 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:09.164454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:09.165818 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:09.167084 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:09.167092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:09.167094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:09.167327 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:09.167546 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:09.167547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:09.167547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:09.167647 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:09.167931 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:09.167939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:09.168101 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.168207 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.168357 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260812 14:31:09.169162 19348 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41375
I20260812 14:31:09.169270 19348 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:09.169457 19348 heartbeater.cc:511] Master 127.23.173.254:41375 requested a full tablet report, sending...
I20260812 14:31:09.169865 18792 ts_manager.cc:194] Registered new tserver with Master: b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479)
I20260812 14:31:09.170243 18792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:43487
W20260812 14:31:09.250108 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:09.250315 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:09.250339 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:09.251461 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:09.251504 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:09.251529 19352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:09.253001 19352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=41939
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:09.253212 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:09.253428 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:09.258404 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:09.258464 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:09.258594 19352 server_base.cc:1034] running on GCE node
W20260812 14:31:09.258706 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:09.258950 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:09.259161 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:09.260300 19352 hybrid_clock.cc:648] HybridClock initialized: now 1786545069260286 us; error 29 us; skew 500 ppm
Aug 12 14:31:09 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545069, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:09.262826 19352 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:31:09.263286 19352 webserver.cc:460] Webserver started at http://127.23.173.195:41939/ using document root <none> and password file <none>
I20260812 14:31:09.263494 19352 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:09.263554 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:09.264914 19352 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:09.265506 19381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.265668 19352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:09.265753 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "685480c9ec4fd341a95eaca468a6d831"
server_key_iv: "ac8eeb9fdfd02fca397baa4265c39b72"
server_key_version: "encryptionkey@0"
I20260812 14:31:09.266103 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:09.294891 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:09.296873 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:09.297043 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:09.297297 19352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:09.297595 19352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:09.297631 19352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.297660 19352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:09.297676 19352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.303292 19352 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:35807
I20260812 14:31:09.303359 19585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:35807 every 8 connection(s)
I20260812 14:31:09.303673 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:31:09.304188 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 19352
Aug 12 14:31:09 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545069, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:09.311604 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.304640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.23.173.254:41375)
0812 14:31:09.304694 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.304696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.304969 (+ 273us) server_negotiation.cc:407] Connection header received
0812 14:31:09.305037 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.305038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.305051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.305107 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:09.305701 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.306477 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.307469 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.307691 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.307796 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.309035 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:09.309037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:09.309037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:09.309044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:09.310426 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.310825 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.310826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.310826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.310873 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.311083 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.311084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.311084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.311213 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:09.311384 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.311423 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.311509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:31:09.314003 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.304727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:58157 (local address 127.23.173.254:41375)
0812 14:31:09.304811 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.304812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.305071 (+ 259us) server_negotiation.cc:407] Connection header received
0812 14:31:09.305592 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.305594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.305607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.305680 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:09.306877 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.307588 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.308806 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.309106 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.309156 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.311529 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:09.311530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:09.311531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:09.311538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:09.312897 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.313254 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.313255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.313255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.313303 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.313485 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.313486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.313486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.313588 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:09.313643 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.313870 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.313928 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":54,"threads_started":1}
I20260812 14:31:09.314157 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.304760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:58157)
0812 14:31:09.304937 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.304987 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.305494 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.305749 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.305753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.306520 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:09.306793 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.306799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.307662 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.307663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.308706 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.308709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.309183 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.309185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.309332 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.309666 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:09.309676 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:09.311400 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:09.312968 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:09.312976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:09.312979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:09.313198 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:09.313348 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:09.313348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:09.313349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:09.313444 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:09.313640 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:09.313646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:09.313770 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.313854 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.313974 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260812 14:31:09.314559 19586 heartbeater.cc:347] Connected to a master server at 127.23.173.254:41375
I20260812 14:31:09.314669 19586 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:09.314815 19586 heartbeater.cc:511] Master 127.23.173.254:41375 requested a full tablet report, sending...
I20260812 14:31:09.315106 18792 ts_manager.cc:194] Registered new tserver with Master: cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
I20260812 14:31:09.315445 18792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:58157
I20260812 14:31:09.323119 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:09.325975 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.323502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.23.173.254:41375)
0812 14:31:09.323564 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.323566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.323664 (+ 98us) server_negotiation.cc:407] Connection header received
0812 14:31:09.323763 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.323764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.323803 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.323862 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:09.323982 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.324446 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.325182 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.325384 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.325440 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.325601 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:09.325647 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 14:31:09.325926 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58746: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260812 14:31:09.326033 19592 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58746: token expired
W20260812 14:31:09.328328 18664 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.173.254:41375)
I20260812 14:31:09.333761 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.328607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.23.173.254:41375)
0812 14:31:09.328653 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.328654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.328661 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:31:09.328681 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.328682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.328691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.328737 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:09.329192 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.329632 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.330274 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.330455 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.330506 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.331596 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:09.331597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:09.331598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:09.331604 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:09.332733 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.333032 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.333033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.333033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.333090 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:09.333248 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:09.333249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:09.333249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:09.333385 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:09.333475 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.333599 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.333675 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:31:09.334851 18792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:09.336548 18792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:09.338580 18999 tablet_service.cc:1467] Processing CreateTablet for tablet ccce82cc4deb4ebe91a941874b145850 (DEFAULT_TABLE table=test-table [id=98feea191e484586af6860b7c32d6b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:09.338771 18999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccce82cc4deb4ebe91a941874b145850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:09.340142 18928 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Bootstrap starting.
I20260812 14:31:09.340607 18928 tablet_bootstrap.cc:654] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:09.341248 18928 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: No bootstrap required, opened a new log
I20260812 14:31:09.341312 18928 ts_tablet_manager.cc:1397] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:09.341488 18928 raft_consensus.cc:348] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.341563 18928 raft_consensus.cc:374] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:09.341622 18928 raft_consensus.cc:729] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53226e7ce9e4ab792894775a396a72a, State: Initialized, Role: FOLLOWER
I20260812 14:31:09.341667 18928 consensus_queue.cc:260] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.341758 18928 ts_tablet_manager.cc:1428] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:09.341922 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.338270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39479 (local address 127.0.0.1:36392)
0812 14:31:09.338303 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.338313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.338345 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.338726 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.338728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.338729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:09.338770 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.338771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.339749 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.339750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.340683 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.340685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.341721 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.341722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.341787 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.341791 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.341817 (+ 26us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.341859 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260812 14:31:09.342352 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.338389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.23.173.194:39479)
0812 14:31:09.338578 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.338583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.338594 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:09.338629 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.338630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.338643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.338720 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:09.338815 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.339661 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.340770 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.341627 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.342145 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.342211 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.342233 (+ 22us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.342273 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260812 14:31:09.342970 19240 tablet_service.cc:1467] Processing CreateTablet for tablet ccce82cc4deb4ebe91a941874b145850 (DEFAULT_TABLE table=test-table [id=98feea191e484586af6860b7c32d6b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:09.343343 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccce82cc4deb4ebe91a941874b145850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:09.344012 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.338361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35807 (local address 127.0.0.1:36852)
0812 14:31:09.339802 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.339811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.339849 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.340675 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.340676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.340678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:09.340715 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.340716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.341941 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.341942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.342883 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.342886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.343787 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.343788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.343837 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.343841 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.343901 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.343944 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":143,"threads_started":1}
I20260812 14:31:09.344033 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.340271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.23.173.195:35807)
0812 14:31:09.340507 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.340511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.340522 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:09.340561 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.340563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.340578 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.340783 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:09.340875 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.341873 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.342967 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.343672 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.343787 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.343846 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.343916 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.343956 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":133,"threads_started":1}
I20260812 14:31:09.344755 19467 tablet_service.cc:1467] Processing CreateTablet for tablet ccce82cc4deb4ebe91a941874b145850 (DEFAULT_TABLE table=test-table [id=98feea191e484586af6860b7c32d6b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:09.345062 19467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccce82cc4deb4ebe91a941874b145850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:09.346369 19611 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Bootstrap starting.
I20260812 14:31:09.346956 19611 tablet_bootstrap.cc:654] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:09.347496 19611 log.cc:824] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:09.347743 19614 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Bootstrap starting.
I20260812 14:31:09.348345 19614 tablet_bootstrap.cc:654] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:09.348395 19611 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: No bootstrap required, opened a new log
I20260812 14:31:09.348451 19611 ts_tablet_manager.cc:1397] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:09.348923 19614 log.cc:824] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:09.349663 19614 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: No bootstrap required, opened a new log
I20260812 14:31:09.349735 19614 ts_tablet_manager.cc:1397] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:09.350349 19611 raft_consensus.cc:348] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.350541 19611 raft_consensus.cc:374] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:09.350581 19611 raft_consensus.cc:729] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45df406c7a442cd80b31f7b68f34c3a, State: Initialized, Role: FOLLOWER
I20260812 14:31:09.350692 19611 consensus_queue.cc:260] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.350977 19611 ts_tablet_manager.cc:1428] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:31:09.350991 19348 heartbeater.cc:503] Master 127.23.173.254:41375 was elected leader, sending a full tablet report...
I20260812 14:31:09.351554 19614 raft_consensus.cc:348] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.351732 19614 raft_consensus.cc:374] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:09.351774 19614 raft_consensus.cc:729] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd04a39700374b35a094d2c5a4ab1f0a, State: Initialized, Role: FOLLOWER
I20260812 14:31:09.351909 19614 consensus_queue.cc:260] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.352183 19614 ts_tablet_manager.cc:1428] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260812 14:31:09.352211 19586 heartbeater.cc:503] Master 127.23.173.254:41375 was elected leader, sending a full tablet report...
W20260812 14:31:09.410209 19350 tablet.cc:2369] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:31:09.544364 19105 tablet.cc:2369] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:31:09.554224 19589 tablet.cc:2369] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:31:09.591482 19110 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:09.591665 19110 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.592077 19110 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
I20260812 14:31:09.595052 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.592589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:47453 (local address 127.23.173.195:35807)
0812 14:31:09.592639 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.592641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.592648 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:31:09.592674 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.592675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.592684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.592739 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:09.592905 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.593365 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.594361 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.594736 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.594801 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.594810 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.594945 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.594988 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:31:09.595059 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.592359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:57767 (local address 127.23.173.194:39479)
0812 14:31:09.592429 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.592431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.592512 (+ 81us) server_negotiation.cc:407] Connection header received
0812 14:31:09.592630 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.592631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.592643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.592696 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:09.592873 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.593359 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.594359 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.594737 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.594803 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.594811 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.594946 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.594983 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:31:09.595079 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.592452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35807 (local address 127.23.173.193:47453)
0812 14:31:09.592537 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.592547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.592601 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.592748 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.592749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.592751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:09.592833 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.592835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.593462 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.593463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.594266 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.594269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.594813 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.594814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.594878 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.594882 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.594928 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.594970 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":17,"thread_start_us":40,"threads_started":1}
I20260812 14:31:09.595072 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.592268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39479 (local address 127.23.173.193:57767)
0812 14:31:09.592407 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.592421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:09.592507 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:09.592719 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:09.592720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:09.592723 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:09.592805 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.592809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.593439 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.593440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.594268 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:09.594270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.594821 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.594822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:09.594880 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.594883 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:09.594934 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:31:09.594981 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":39,"thread_start_us":73,"threads_started":1}
I20260812 14:31:09.595472 19498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccce82cc4deb4ebe91a941874b145850" candidate_uuid: "c53226e7ce9e4ab792894775a396a72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" is_pre_election: true
I20260812 14:31:09.595597 19498 raft_consensus.cc:2454] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c53226e7ce9e4ab792894775a396a72a in term 0.
I20260812 14:31:09.595572 19268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccce82cc4deb4ebe91a941874b145850" candidate_uuid: "c53226e7ce9e4ab792894775a396a72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45df406c7a442cd80b31f7b68f34c3a" is_pre_election: true
I20260812 14:31:09.595685 19268 raft_consensus.cc:2454] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c53226e7ce9e4ab792894775a396a72a in term 0.
I20260812 14:31:09.595826 18918 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c53226e7ce9e4ab792894775a396a72a, cd04a39700374b35a094d2c5a4ab1f0a; no voters:
I20260812 14:31:09.595953 19110 raft_consensus.cc:2781] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:09.596028 19110 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:09.596045 19110 raft_consensus.cc:3037] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:09.596752 19110 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.596866 19110 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 1 election: Requested vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
I20260812 14:31:09.597055 19268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccce82cc4deb4ebe91a941874b145850" candidate_uuid: "c53226e7ce9e4ab792894775a396a72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45df406c7a442cd80b31f7b68f34c3a"
I20260812 14:31:09.597055 19498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccce82cc4deb4ebe91a941874b145850" candidate_uuid: "c53226e7ce9e4ab792894775a396a72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
I20260812 14:31:09.597122 19268 raft_consensus.cc:3037] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:09.597124 19498 raft_consensus.cc:3037] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:09.597790 19268 raft_consensus.cc:2454] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c53226e7ce9e4ab792894775a396a72a in term 1.
I20260812 14:31:09.597877 19498 raft_consensus.cc:2454] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c53226e7ce9e4ab792894775a396a72a in term 1.
I20260812 14:31:09.597961 18918 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45df406c7a442cd80b31f7b68f34c3a, c53226e7ce9e4ab792894775a396a72a; no voters:
I20260812 14:31:09.598057 19110 raft_consensus.cc:2781] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:09.598178 19110 raft_consensus.cc:686] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 LEADER]: Becoming Leader. State: Replica: c53226e7ce9e4ab792894775a396a72a, State: Running, Role: LEADER
I20260812 14:31:09.598244 19110 consensus_queue.cc:237] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:09.598959 18792 catalog_manager.cc:5495] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a reported cstate change: term changed from 0 to 1, leader changed from <none> to c53226e7ce9e4ab792894775a396a72a (127.23.173.193). New cstate: current_term: 1 leader_uuid: "c53226e7ce9e4ab792894775a396a72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } health_report { overall_health: HEALTHY } } }
I20260812 14:31:09.624822 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.622112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.23.173.193:43287)
0812 14:31:09.622239 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:09.622241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:09.622249 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:31:09.622289 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:09.622289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:09.622307 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:09.622377 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:09.622475 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.623533 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.624206 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:09.624407 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:09.624452 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:09.624563 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:09.624660 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:09.624711 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:09.624729 (+ 18us) server_negotiation.cc:299] Negotiation successful
0812 14:31:09.624743 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260812 14:31:09.626247 19498 raft_consensus.cc:1261] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Refusing update from remote peer c53226e7ce9e4ab792894775a396a72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:09.626266 19268 raft_consensus.cc:1261] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Refusing update from remote peer c53226e7ce9e4ab792894775a396a72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:09.626552 19110 consensus_queue.cc:1035] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:09.626685 19629 consensus_queue.cc:1035] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:09.628937 19639 mvcc.cc:205] Tried to move back new op lower bound from 7317688605186883584 to 7317688605074948096. Current Snapshot: MvccSnapshot[applied={T|T < 7317688605186883584}]
I20260812 14:31:09.629143 19640 mvcc.cc:205] Tried to move back new op lower bound from 7317688605186883584 to 7317688605074948096. Current Snapshot: MvccSnapshot[applied={T|T < 7317688605186883584}]
I20260812 14:31:09.629726 19641 mvcc.cc:205] Tried to move back new op lower bound from 7317688605186883584 to 7317688605074948096. Current Snapshot: MvccSnapshot[applied={T|T < 7317688605186883584}]
I20260812 14:31:09.630957 18787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:09.631060 18787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:09.631786 18787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Sending DeleteTablet for 3 replicas of tablet ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:09.632073 18999 tablet_service.cc:1514] Processing DeleteTablet for tablet ccce82cc4deb4ebe91a941874b145850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:09 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36086
I20260812 14:31:09.632166 19240 tablet_service.cc:1514] Processing DeleteTablet for tablet ccce82cc4deb4ebe91a941874b145850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:09 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36392
I20260812 14:31:09.632177 19467 tablet_service.cc:1514] Processing DeleteTablet for tablet ccce82cc4deb4ebe91a941874b145850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:09 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:36852
I20260812 14:31:09.632381 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 18853
I20260812 14:31:09.642057 19651 tablet_replica.cc:330] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: stopping tablet replica
I20260812 14:31:09.642184 19651 raft_consensus.cc:2229] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:09.642294 19651 raft_consensus.cc:2258] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:09.643664 19651 ts_tablet_manager.cc:1905] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:09.644938 19653 tablet_replica.cc:330] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: stopping tablet replica
I20260812 14:31:09.645040 19653 raft_consensus.cc:2229] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:09.645146 19653 raft_consensus.cc:2258] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:09.645192 19651 ts_tablet_manager.cc:1918] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:09.645364 19651 log.cc:1196] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:09.645550 19653 ts_tablet_manager.cc:1905] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:09.645651 19651 ts_tablet_manager.cc:1939] T ccce82cc4deb4ebe91a941874b145850 P cd04a39700374b35a094d2c5a4ab1f0a: Deleting consensus metadata
I20260812 14:31:09.646137 18765 catalog_manager.cc:4841] TS cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): tablet ccce82cc4deb4ebe91a941874b145850 (table test-table [id=98feea191e484586af6860b7c32d6b65]) successfully deleted
I20260812 14:31:09.646842 19653 ts_tablet_manager.cc:1918] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:09.646903 19653 log.cc:1196] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:09.647092 19653 ts_tablet_manager.cc:1939] T ccce82cc4deb4ebe91a941874b145850 P b45df406c7a442cd80b31f7b68f34c3a: Deleting consensus metadata
I20260812 14:31:09.647495 18765 catalog_manager.cc:4841] TS b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): tablet ccce82cc4deb4ebe91a941874b145850 (table test-table [id=98feea191e484586af6860b7c32d6b65]) successfully deleted
W20260812 14:31:09.647845 18764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:43287 (error 108)
I20260812 14:31:09.648283 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:09.648183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:43287 (local address 127.0.0.1:36096)
0812 14:31:09.648222 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:09.648255 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260812 14:31:09.648418 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 19106
W20260812 14:31:09.648401 18764 catalog_manager.cc:4584] TS c53226e7ce9e4ab792894775a396a72a (127.23.173.193:43287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccce82cc4deb4ebe91a941874b145850: Network error: Client connection negotiation failed: client connection to 127.23.173.193:43287: connect: Connection refused (error 111)
I20260812 14:31:09.661885 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 19352
I20260812 14:31:09.670805 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 18733
I20260812 14:31:29.680290 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41375
--webserver_interface=127.23.173.254
--webserver_port=33835
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:29.763147 19658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:29.763363 19658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:29.763386 19658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:29.764614 19658 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:31:29.764662 19658 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:31:29.764678 19658 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:31:29.764693 19658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:29.764719 19658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:29.764736 19658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:29.764758 19658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:29.764771 19658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:29.766551 19658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:29.766765 19658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:29.767001 19658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:29.772214 19674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:29.772267 19677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:29.772202 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:29.772365 19658 server_base.cc:1034] running on GCE node
I20260812 14:31:29.772720 19658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:29.773025 19658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:29.774173 19658 hybrid_clock.cc:648] HybridClock initialized: now 1786545089774160 us; error 29 us; skew 500 ppm
Aug 12 14:31:29 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545089, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:29.776832 19658 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:31:29.777258 19658 webserver.cc:460] Webserver started at http://127.23.173.254:33835/ using document root <none> and password file <none>
I20260812 14:31:29.777480 19658 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:29.777532 19658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:29.778848 19658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:31:29.779524 19691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:29.779727 19658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:31:29.779825 19658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "be698f77bec4420cab0540d3fcdd86aa"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20959e9242b42cb8ba8604d6bc6de0d7"
server_key_iv: "4541c1792eaa66fbc1c0dd5d6f0a2b5a"
server_key_version: "encryptionkey@0"
I20260812 14:31:29.780181 19658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:29.823868 19658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:29.826164 19658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:29.826326 19658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:29.830531 19658 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:41375
I20260812 14:31:29.830569 19791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:41375 every 8 connection(s)
I20260812 14:31:29.830955 19658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:29.832528 19793 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:31:29.833204 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap starting.
I20260812 14:31:29.836773 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 19658
I20260812 14:31:29.837087 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:43287
--local_ip_for_outbound_sockets=127.23.173.193
--tserver_master_addrs=127.23.173.254:41375
--webserver_port=34847
--webserver_interface=127.23.173.193
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:31:29.841884 19793 log.cc:824] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:29.844238 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:31:29.844534 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Bootstrap complete.
I20260812 14:31:29.846652 19793 raft_consensus.cc:348] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:29.846876 19793 raft_consensus.cc:729] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: be698f77bec4420cab0540d3fcdd86aa, State: Initialized, Role: FOLLOWER
I20260812 14:31:29.847033 19793 consensus_queue.cc:260] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:29.847115 19793 raft_consensus.cc:388] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:29.847168 19793 raft_consensus.cc:482] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:29.847216 19793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:31:29.848289 19793 raft_consensus.cc:504] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:29.848387 19793 leader_election.cc:302] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be698f77bec4420cab0540d3fcdd86aa; no voters:
I20260812 14:31:29.848558 19793 leader_election.cc:288] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 14:31:29.848630 19798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:31:29.848827 19793 sys_catalog.cc:564] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:29.848855 19798 raft_consensus.cc:686] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [term 3 LEADER]: Becoming Leader. State: Replica: be698f77bec4420cab0540d3fcdd86aa, State: Running, Role: LEADER
I20260812 14:31:29.848966 19798 consensus_queue.cc:237] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } }
I20260812 14:31:29.849280 19798 sys_catalog.cc:455] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader be698f77bec4420cab0540d3fcdd86aa. Latest consensus state: current_term: 3 leader_uuid: "be698f77bec4420cab0540d3fcdd86aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } } }
I20260812 14:31:29.849354 19798 sys_catalog.cc:458] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:29.849282 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "be698f77bec4420cab0540d3fcdd86aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be698f77bec4420cab0540d3fcdd86aa" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 41375 } } }
I20260812 14:31:29.849854 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:29.849996 19811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:29.850875 19811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:29.850996 19811 catalog_manager.cc:1259] Loaded cluster ID: 82d000c90b654ff0bd46aa81caeea6d3
I20260812 14:31:29.851028 19811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:29.852249 19811 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:29.852425 19811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Loaded TSK: 0
I20260812 14:31:29.852469 19811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be698f77bec4420cab0540d3fcdd86aa: Loaded TSK: 1
F20260812 14:31:29.852710 19811 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2aa50a99ad google::LogMessage::Fail() at ??:0
@ 0x7f2aa50ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2aa50a94de google::LogMessage::Flush() at ??:0
@ 0x7f2aa50aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2aa7816104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2aa7813d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2aad0031cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2aad00348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2aad003594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2aacfc9c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2aacfd3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2aa6f770d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2aa6f6c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2aa7dd06db start_thread at ??:0
@ 0x7f2aa3ba761f clone at ??:0
W20260812 14:31:29.938309 19795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:29.938498 19795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:29.938519 19795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:29.939618 19795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:29.939658 19795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:29.939682 19795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:29.941097 19795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:43287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=34847
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:29.941293 19795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:29.941514 19795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:29.946470 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:29.946466 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:29.946465 19859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:29.946614 19795 server_base.cc:1034] running on GCE node
I20260812 14:31:29.947046 19795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:29.947250 19795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:29.948387 19795 hybrid_clock.cc:648] HybridClock initialized: now 1786545089948374 us; error 28 us; skew 500 ppm
Aug 12 14:31:29 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545089, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:29.951140 19795 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:31:29.951622 19795 webserver.cc:460] Webserver started at http://127.23.173.193:34847/ using document root <none> and password file <none>
I20260812 14:31:29.951895 19795 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:29.951949 19795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:29.953261 19795 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:29.953909 19882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:29.954118 19795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:29.954213 19795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "c53226e7ce9e4ab792894775a396a72a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "20c6037a208c4e95614e854ac44415df"
server_key_iv: "22d89131b0e6308ffba65bd8d196ac5b"
server_key_version: "encryptionkey@0"
I20260812 14:31:29.954653 19795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:29.969313 19795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:29.971657 19795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:29.971835 19795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:29.972102 19795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:29.972607 19894 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:31:29.974054 19795 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:31:29.974118 19795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:31:29.974150 19795 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:31:29.975021 19795 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:31:29.975070 19795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:31:29.975121 19894 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Bootstrap starting.
I20260812 14:31:29.982491 19795 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:43287
I20260812 14:31:29.982641 20079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:43287 every 8 connection(s)
I20260812 14:31:29.983099 19795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:31:29.983280 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 19795
I20260812 14:31:29.983625 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:39479
--local_ip_for_outbound_sockets=127.23.173.194
--tserver_master_addrs=127.23.173.254:41375
--webserver_port=45887
--webserver_interface=127.23.173.194
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:31:29.986270 19894 log.cc:824] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:29.988364 19894 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:31:29.988672 19894 tablet_bootstrap.cc:492] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Bootstrap complete.
I20260812 14:31:29.989790 19894 ts_tablet_manager.cc:1397] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260812 14:31:29.991240 19894 raft_consensus.cc:348] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:29.991575 19894 raft_consensus.cc:729] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53226e7ce9e4ab792894775a396a72a, State: Initialized, Role: FOLLOWER
I20260812 14:31:29.991791 19894 consensus_queue.cc:260] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:29.992087 19894 ts_tablet_manager.cc:1428] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260812 14:31:30.047236 20082 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:31:29.983902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:56641)
0812 14:31:29.984125 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:29.984179 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:29.984780 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:30.047019 (+ 62239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
W20260812 14:31:30.047542 20080 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:41375 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.254:41375: Client connection negotiation failed: client connection to 127.23.173.254:41375: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:31:30.047928 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.047793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:35877)
0812 14:31:30.047848 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.047900 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:31:30.048271 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.048180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:47729)
0812 14:31:30.048225 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.048252 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:31:30.048516 20080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:31:30.070338 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:30.070539 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:30.070561 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:30.071686 20083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:30.071753 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:30.071786 20083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:30.073230 20083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:39479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=45887
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:30.073450 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:30.073681 20083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:30.078507 20101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:30.078511 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:30.078560 20083 server_base.cc:1034] running on GCE node
W20260812 14:31:30.078509 20097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:30.078900 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:30.079128 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:30.080265 20083 hybrid_clock.cc:648] HybridClock initialized: now 1786545090080250 us; error 30 us; skew 500 ppm
Aug 12 14:31:30 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545090, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:30.082696 20083 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:31:30.083115 20083 webserver.cc:460] Webserver started at http://127.23.173.194:45887/ using document root <none> and password file <none>
I20260812 14:31:30.083313 20083 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:30.083364 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:30.084656 20083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:30.085211 20116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:30.085367 20083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:30.085469 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "b45df406c7a442cd80b31f7b68f34c3a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "0feebdce9e6647b25d0511116644c144"
server_key_iv: "b433af20c977aabd852d7f602a52dec4"
server_key_version: "encryptionkey@0"
I20260812 14:31:30.085777 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:30.117408 20083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:30.119285 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:30.119448 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:30.119692 20083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:30.120030 20083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:30.120064 20083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:30.120096 20083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:30.120113 20083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:30.125823 20083 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:39479
I20260812 14:31:30.125883 20319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:39479 every 8 connection(s)
I20260812 14:31:30.126199 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:31:30.127187 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.126823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:53295)
0812 14:31:30.126970 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.127043 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20260812 14:31:30.127597 20320 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:41375 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.254:41375: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
I20260812 14:31:30.127887 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.127799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:43157)
0812 14:31:30.127840 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.127868 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 14:31:30.128111 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.128046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:50641)
0812 14:31:30.128076 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.128098 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260812 14:31:30.128189 20320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:31:30.134377 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 20083
I20260812 14:31:30.134645 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:35807
--local_ip_for_outbound_sockets=127.23.173.195
--tserver_master_addrs=127.23.173.254:41375
--webserver_port=41939
--webserver_interface=127.23.173.195
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:30.218387 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:30.218603 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:30.218627 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:30.219794 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:30.219834 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:30.219859 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:30.221318 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=41939
--tserver_master_addrs=127.23.173.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:30.221535 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:30.221760 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:30.226645 20332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:30.226672 20335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:30.226662 20331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:30.226858 20324 server_base.cc:1034] running on GCE node
I20260812 14:31:30.227121 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:30.227370 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:30.228519 20324 hybrid_clock.cc:648] HybridClock initialized: now 1786545090228503 us; error 31 us; skew 500 ppm
Aug 12 14:31:30 dist-test-slave-1l3l krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545090, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:30.230938 20324 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:31:30.231388 20324 webserver.cc:460] Webserver started at http://127.23.173.195:41939/ using document root <none> and password file <none>
I20260812 14:31:30.231586 20324 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:30.231635 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:30.232924 20324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:31:30.233470 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:30.233633 20324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:30.233714 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "cd04a39700374b35a094d2c5a4ab1f0a"
format_stamp: "Formatted at 2026-08-12 14:30:48 on dist-test-slave-1l3l"
server_key: "685480c9ec4fd341a95eaca468a6d831"
server_key_iv: "ac8eeb9fdfd02fca397baa4265c39b72"
server_key_version: "encryptionkey@0"
I20260812 14:31:30.234063 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:30.242527 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:30.244436 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:30.244572 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:30.244771 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:30.245070 20324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:30.245105 20324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:30.245134 20324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:30.245160 20324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:30.250876 20324 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:35807
I20260812 14:31:30.250932 20551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:35807 every 8 connection(s)
I20260812 14:31:30.251262 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:31:30.252276 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.251913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:35191)
0812 14:31:30.252064 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.252137 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260812 14:31:30.252684 20552 heartbeater.cc:644] Failed to heartbeat to 127.23.173.254:41375 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.254:41375: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
I20260812 14:31:30.252967 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.252878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:49741)
0812 14:31:30.252921 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.252949 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:31:30.253228 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:30.253170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:39665)
0812 14:31:30.253195 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:30.253214 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260812 14:31:30.253358 20552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:31:30.260419 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 20324
I20260812 14:31:31.049131 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.048876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:42219)
0812 14:31:31.049009 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:31.049048 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260812 14:31:31.128698 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.128460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:36763)
0812 14:31:31.128607 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:31.128644 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260812 14:31:31.253901 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.253689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:50077)
0812 14:31:31.253817 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:31.253850 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260812 14:31:31.863387 20560 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:31.863533 20560 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:31.863889 20560 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
Aug 12 14:31:31 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545089, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.195@KRBTEST.COM
Aug 12 14:31:31 dist-test-slave-1l3l krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545089, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.194@KRBTEST.COM
I20260812 14:31:31.874089 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.864150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51725 (local address 127.23.173.194:39479)
0812 14:31:31.864320 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:31.864334 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:31.864359 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:31:31.864806 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:31.864807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:31.864850 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:31.864972 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:31.865794 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.866671 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.867670 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.868112 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.868213 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:31.870813 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:31.870816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:31.870816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:31.870826 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:31.872629 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:31.872986 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:31.872989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:31.872989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:31.873113 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:31.873321 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:31.873322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:31.873322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:31.873574 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:31.873653 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:31.873851 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 14:31:31.873972 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260812 14:31:31.874078 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.864196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:35807 (local address 127.23.173.193:54983)
0812 14:31:31.864277 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:31.864288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:31.864338 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:31.865069 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:31.865071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:31.865459 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:31.865663 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:31.865669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.866761 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.866762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:31.867576 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:31.867579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.868200 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.868201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:31.868310 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:31.868851 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:31.868859 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:31.870286 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:31.872502 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:31.872509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:31.872510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:31.872724 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:31.872992 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:31.872993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:31.872993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:31.873135 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:31.873548 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:31.873555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:31.873693 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 14:31:31.873781 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:31:31.873984 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":35,"thread_start_us":34,"threads_started":1}
I20260812 14:31:31.874099 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.864325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:54983 (local address 127.23.173.195:35807)
0812 14:31:31.864451 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:31.864462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:31.864480 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:31:31.864891 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:31.864893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:31.864935 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:31.865075 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:31.865793 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.866661 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.867668 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.868115 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.868217 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:31.870400 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:31.870404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:31.870404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:31.870417 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:31.872413 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:31.872804 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:31.872807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:31.872808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:31.872920 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:31.873197 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:31.873198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:31.873198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:31.873481 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:31.873540 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:31.873856 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 14:31:31.873979 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260812 14:31:31.874181 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:31.864040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:39479 (local address 127.23.173.193:51725)
0812 14:31:31.864146 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:31.864156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:31.864225 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:31.865040 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:31.865044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:31.865467 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:31.865663 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:31.865669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.866782 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.866783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:31.867578 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:31.867580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:31.868197 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:31.868198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:31.868311 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:31.868902 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:31.868907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:31.870687 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:31.872698 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:31.872699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:31.872699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:31.872901 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:31.873186 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:31.873187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:31.873188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:31.873258 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:31.873643 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:31.873644 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:31.873696 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 14:31:31.873782 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:31:31.874059 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":192,"thread_start_us":48,"threads_started":1}
W20260812 14:31:31.875084 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
W20260812 14:31:31.875160 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:31.875177 19885 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c53226e7ce9e4ab792894775a396a72a; no voters: b45df406c7a442cd80b31f7b68f34c3a, cd04a39700374b35a094d2c5a4ab1f0a
I20260812 14:31:31.875272 20560 raft_consensus.cc:2726] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:31:32.049713 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:32.049570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:54637)
0812 14:31:32.049630 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:32.049672 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:31:32.129423 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:32.129175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:59331)
0812 14:31:32.129294 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:32.129373 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260812 14:31:32.254534 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:32.254282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:49757)
0812 14:31:32.254436 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:32.254486 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260812 14:31:33.050333 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:33.050113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:53685)
0812 14:31:33.050247 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:33.050283 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260812 14:31:33.130149 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:33.129912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:50589)
0812 14:31:33.130059 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:33.130104 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260812 14:31:33.255192 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:33.254944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:52191)
0812 14:31:33.255107 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:33.255149 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260812 14:31:34.050946 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:34.050746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:38133)
0812 14:31:34.050863 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:34.050901 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260812 14:31:34.059211 20581 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:34.059288 20581 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:34.059435 20581 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
W20260812 14:31:34.059868 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
W20260812 14:31:34.059935 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:34.059959 19885 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c53226e7ce9e4ab792894775a396a72a; no voters: b45df406c7a442cd80b31f7b68f34c3a, cd04a39700374b35a094d2c5a4ab1f0a
I20260812 14:31:34.060038 20581 raft_consensus.cc:2726] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:31:34.130782 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:34.130552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:56575)
0812 14:31:34.130692 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:34.130725 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260812 14:31:34.255834 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:34.255596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:36205)
0812 14:31:34.255744 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:34.255777 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260812 14:31:35.051668 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:35.051406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:41209)
0812 14:31:35.051563 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:35.051619 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260812 14:31:35.131469 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:35.131230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:51969)
0812 14:31:35.131381 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:35.131423 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260812 14:31:35.256490 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:35.256258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:60253)
0812 14:31:35.256404 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:35.256445 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260812 14:31:36.052350 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:36.052129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:47573)
0812 14:31:36.052261 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:36.052300 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260812 14:31:36.132089 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:36.131877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:44005)
0812 14:31:36.132001 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:36.132044 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260812 14:31:36.257148 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:36.256919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:43481)
0812 14:31:36.257060 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:36.257101 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260812 14:31:36.763830 20592 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:36.763924 20592 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:36.764060 20592 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
W20260812 14:31:36.764444 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
W20260812 14:31:36.764509 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:36.764523 19885 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c53226e7ce9e4ab792894775a396a72a; no voters: b45df406c7a442cd80b31f7b68f34c3a, cd04a39700374b35a094d2c5a4ab1f0a
I20260812 14:31:36.764600 20592 raft_consensus.cc:2726] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:31:37.053038 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:37.052820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:44255)
0812 14:31:37.052953 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:37.052991 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260812 14:31:37.132714 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:37.132497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:44303)
0812 14:31:37.132635 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:37.132670 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260812 14:31:37.257774 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:37.257535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:37247)
0812 14:31:37.257696 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:37.257733 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260812 14:31:38.053663 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:38.053452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:44691)
0812 14:31:38.053576 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:38.053615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260812 14:31:38.133385 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:38.133181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:37809)
0812 14:31:38.133303 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:38.133341 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260812 14:31:38.258438 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:38.258210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:45517)
0812 14:31:38.258342 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:38.258385 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260812 14:31:39.054293 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:39.054057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:45945)
0812 14:31:39.054204 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:39.054243 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260812 14:31:39.134109 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:39.133873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:48131)
0812 14:31:39.134012 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:39.134060 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260812 14:31:39.259148 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:39.258924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:60471)
0812 14:31:39.259070 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:39.259106 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260812 14:31:40.054919 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:40.054705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:45615)
0812 14:31:40.054833 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:40.054870 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260812 14:31:40.134752 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:40.134551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:41373)
0812 14:31:40.134672 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:40.134708 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260812 14:31:40.140064 20612 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:40.140146 20612 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:40.140275 20612 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
W20260812 14:31:40.140616 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
W20260812 14:31:40.140695 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:40.140713 19885 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c53226e7ce9e4ab792894775a396a72a; no voters: b45df406c7a442cd80b31f7b68f34c3a, cd04a39700374b35a094d2c5a4ab1f0a
I20260812 14:31:40.140784 20612 raft_consensus.cc:2726] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:31:40.259795 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:40.259534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:43061)
0812 14:31:40.259680 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:40.259743 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260812 14:31:41.055665 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:41.055413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:38371)
0812 14:31:41.055583 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:41.055617 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260812 14:31:41.135591 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:41.135254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:46305)
0812 14:31:41.135482 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:41.135535 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260812 14:31:41.260586 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:41.260306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:32807)
0812 14:31:41.260482 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:41.260534 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260812 14:31:42.056309 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:42.056115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:51273)
0812 14:31:42.056227 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:42.056263 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260812 14:31:42.136447 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:42.136186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:50173)
0812 14:31:42.136360 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:42.136401 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260812 14:31:42.261206 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:42.261007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:42913)
0812 14:31:42.261125 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:42.261161 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260812 14:31:42.438750 20623 raft_consensus.cc:482] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:42.438841 20623 raft_consensus.cc:504] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45df406c7a442cd80b31f7b68f34c3a" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 39479 } } peers { permanent_uuid: "cd04a39700374b35a094d2c5a4ab1f0a" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 35807 } } peers { permanent_uuid: "c53226e7ce9e4ab792894775a396a72a" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 43287 } }
I20260812 14:31:42.439007 20623 leader_election.cc:288] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479), cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807)
W20260812 14:31:42.439440 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b45df406c7a442cd80b31f7b68f34c3a (127.23.173.194:39479): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
W20260812 14:31:42.439518 19885 leader_election.cc:341] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd04a39700374b35a094d2c5a4ab1f0a (127.23.173.195:35807): Not found: Tablet not found: ccce82cc4deb4ebe91a941874b145850
I20260812 14:31:42.439533 19885 leader_election.cc:302] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c53226e7ce9e4ab792894775a396a72a; no voters: b45df406c7a442cd80b31f7b68f34c3a, cd04a39700374b35a094d2c5a4ab1f0a
I20260812 14:31:42.439644 20623 raft_consensus.cc:2726] T ccce82cc4deb4ebe91a941874b145850 P c53226e7ce9e4ab792894775a396a72a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:31:43.056914 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:43.056711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:37337)
0812 14:31:43.056830 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:43.056866 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260812 14:31:43.137140 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:43.136929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:51065)
0812 14:31:43.137048 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:43.137095 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260812 14:31:43.261808 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:43.261588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:50153)
0812 14:31:43.261728 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:43.261764 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260812 14:31:44.057714 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:44.057436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:52103)
0812 14:31:44.057612 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:44.057665 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260812 14:31:44.137836 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:44.137559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:46299)
0812 14:31:44.137742 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:44.137787 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260812 14:31:44.262418 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:44.262209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:37797)
0812 14:31:44.262331 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:44.262370 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260812 14:31:45.058535 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:45.058284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.193:35893)
0812 14:31:45.058446 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:45.058481 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260812 14:31:45.138592 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:45.138392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.194:54861)
0812 14:31:45.138517 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:45.138549 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260812 14:31:45.263096 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:45.262873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:41375 (local address 127.23.173.195:55559)
0812 14:31:45.263012 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:45.263051 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.254:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260812 14:31:45.263515 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 19795
I20260812 14:31:45.274588 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 20083
I20260812 14:31:45.283527 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 20324
2026-08-12T14:31:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58644 ms)
[----------] 30 tests from SecurityITest (265288 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:45 dist-test-slave-1l3l krb5kdc[20643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:45 dist-test-slave-1l3l krb5kdc[20643](info): set up 2 sockets
Aug 12 14:31:45 dist-test-slave-1l3l krb5kdc[20643](info): commencing operation
krb5kdc: starting...
W20260812 14:31:46.465229 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:46 dist-test-slave-1l3l krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:31:46.584692 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37013
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:46.665035 20672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:46.665236 20672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:46.665256 20672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:46.666356 20672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:46.666396 20672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:46.666409 20672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:46.666420 20672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:46.666431 20672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:46.667891 20672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:46.668118 20672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:46.668334 20672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:46.673010 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:46.673025 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:46.673089 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:46.673162 20672 server_base.cc:1034] running on GCE node
I20260812 14:31:46.673514 20672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:46.673791 20672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:46.674932 20672 hybrid_clock.cc:648] HybridClock initialized: now 1786545106674921 us; error 30 us; skew 500 ppm
Aug 12 14:31:46 dist-test-slave-1l3l krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:46.677366 20672 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:31:46.677786 20672 webserver.cc:460] Webserver started at http://127.23.173.254:44309/ using document root <none> and password file <none>
I20260812 14:31:46.677994 20672 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:46.678042 20672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:46.678175 20672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:46.679201 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "cd3a3e9f768f473999c086c8f68b6b9f"
format_stamp: "Formatted at 2026-08-12 14:31:46 on dist-test-slave-1l3l"
server_key: "96e5ee229ad749ecaa6352661d77e411"
server_key_iv: "742c43c6baf562f06501c2f0657674bc"
server_key_version: "encryptionkey@0"
I20260812 14:31:46.679515 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "cd3a3e9f768f473999c086c8f68b6b9f"
format_stamp: "Formatted at 2026-08-12 14:31:46 on dist-test-slave-1l3l"
server_key: "96e5ee229ad749ecaa6352661d77e411"
server_key_iv: "742c43c6baf562f06501c2f0657674bc"
server_key_version: "encryptionkey@0"
I20260812 14:31:46.680804 20672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:31:46.681491 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:46.681645 20672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:46.681727 20672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "cd3a3e9f768f473999c086c8f68b6b9f"
format_stamp: "Formatted at 2026-08-12 14:31:46 on dist-test-slave-1l3l"
server_key: "96e5ee229ad749ecaa6352661d77e411"
server_key_iv: "742c43c6baf562f06501c2f0657674bc"
server_key_version: "encryptionkey@0"
I20260812 14:31:46.681787 20672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:46.690619 20672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:46.692636 20672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:46.692776 20672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:46.696645 20672 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:37013
I20260812 14:31:46.696687 20784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:37013 every 8 connection(s)
I20260812 14:31:46.696985 20672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:46.697803 20785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:46.699146 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 20672
I20260812 14:31:46.699227 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:46.699407 24247 external_mini_cluster.cc:1428] Setting key bccfc408b0fd63c68049784c375dce3b
I20260812 14:31:46.700294 20785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f: Bootstrap starting.
I20260812 14:31:46.700978 20785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:46.701278 20785 log.cc:824] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:46.701996 20785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f: No bootstrap required, opened a new log
I20260812 14:31:46.704205 20785 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37013 } }
I20260812 14:31:46.704377 20785 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:46.704453 20785 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3a3e9f768f473999c086c8f68b6b9f, State: Initialized, Role: FOLLOWER
I20260812 14:31:46.704587 20785 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37013 } }
I20260812 14:31:46.704665 20785 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:46.704689 20785 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:46.704732 20785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:46.705253 20785 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37013 } }
I20260812 14:31:46.705340 20785 leader_election.cc:302] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd3a3e9f768f473999c086c8f68b6b9f; no voters:
I20260812 14:31:46.705494 20785 leader_election.cc:288] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:46.705520 20790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:46.705637 20790 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [term 1 LEADER]: Becoming Leader. State: Replica: cd3a3e9f768f473999c086c8f68b6b9f, State: Running, Role: LEADER
I20260812 14:31:46.705770 20785 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:46.705770 20790 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37013 } }
Aug 12 14:31:46 dist-test-slave-1l3l krb5kdc[20643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:46.706228 20791 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3a3e9f768f473999c086c8f68b6b9f. Latest consensus state: current_term: 1 leader_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37013 } } }
I20260812 14:31:46.706310 20791 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:46.706522 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3a3e9f768f473999c086c8f68b6b9f" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37013 } } }
I20260812 14:31:46.706841 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:46.707016 20803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:46.707580 20803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:46.709635 20803 catalog_manager.cc:1347] Generated new cluster ID: 793282d506ec48c69cb900097db465f7
I20260812 14:31:46.709687 20803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:46.710371 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:46.699943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.23.173.254:37013)
0812 14:31:46.700128 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:46.700138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:46.700169 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:31:46.700690 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:46.700700 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:46.700772 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:46.700923 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:46.701514 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:46.702841 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:46.703587 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:46.704243 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:46.704364 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:46.706887 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:46.706891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:46.706892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:46.706903 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:46.708787 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:46.709201 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:46.709203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:46.709203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:46.709284 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:46.709467 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:46.709468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:46.709468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:46.709756 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:46.709902 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:46.710041 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:31:46.710176 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260812 14:31:46.717640 20803 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:46.718101 20803 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:46.725960 20803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f: Generated new TSK 0
I20260812 14:31:46.726102 20803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:31:46.752524 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37013
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:46.833278 20830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:46.833474 20830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:46.833495 20830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:46.834595 20830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:46.834637 20830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:46.834663 20830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:46.836138 20830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:46.836354 20830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:46.836560 20830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:46.841364 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:46.841374 20838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:46.841455 20839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:46.841562 20830 server_base.cc:1034] running on GCE node
I20260812 14:31:46.841828 20830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:46.842063 20830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:46.843214 20830 hybrid_clock.cc:648] HybridClock initialized: now 1786545106843198 us; error 31 us; skew 500 ppm
Aug 12 14:31:46 dist-test-slave-1l3l krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:46.845750 20830 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:31:46.846143 20830 webserver.cc:460] Webserver started at http://127.23.173.193:44771/ using document root <none> and password file <none>
I20260812 14:31:46.846369 20830 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:46.846429 20830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:46.846566 20830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:46.847961 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "dc550cd46432430795ee68d1c176ccb4"
format_stamp: "Formatted at 2026-08-12 14:31:46 on dist-test-slave-1l3l"
server_key: "0b33779163f159ee889e81027f35670a"
server_key_iv: "82261e0141b7d1f4150b9789abca4688"
server_key_version: "encryptionkey@0"
I20260812 14:31:46.848383 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "dc550cd46432430795ee68d1c176ccb4"
format_stamp: "Formatted at 2026-08-12 14:31:46 on dist-test-slave-1l3l"
server_key: "0b33779163f159ee889e81027f35670a"
server_key_iv: "82261e0141b7d1f4150b9789abca4688"
server_key_version: "encryptionkey@0"
I20260812 14:31:46.849864 20830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:46.850524 20859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:46.850677 20830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:46.850760 20830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "dc550cd46432430795ee68d1c176ccb4"
format_stamp: "Formatted at 2026-08-12 14:31:46 on dist-test-slave-1l3l"
server_key: "0b33779163f159ee889e81027f35670a"
server_key_iv: "82261e0141b7d1f4150b9789abca4688"
server_key_version: "encryptionkey@0"
I20260812 14:31:46.850827 20830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:46.870965 20830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:46.873214 20830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:46.873368 20830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:46.873597 20830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:46.873905 20830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:46.873940 20830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:46.873970 20830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:46.873988 20830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:46.879731 20830 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:35877
I20260812 14:31:46.879774 21085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:35877 every 8 connection(s)
I20260812 14:31:46.880090 20830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:31:46 dist-test-slave-1l3l krb5kdc[20643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:46.887684 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 20830
I20260812 14:31:46.887812 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:31:46.888036 24247 external_mini_cluster.cc:1428] Setting key 21195dbb49db73c4a2b4ab28551f4d20
I20260812 14:31:46.888896 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:46.880758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:36857 (local address 127.23.173.254:37013)
0812 14:31:46.880825 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:46.880827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:46.881019 (+ 192us) server_negotiation.cc:407] Connection header received
0812 14:31:46.881479 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:46.881480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:46.881489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:46.881539 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:46.882439 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:46.882946 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:46.883944 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:46.884171 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:46.884219 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:46.886203 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:46.886204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:46.886204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:46.886211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:46.887460 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:46.887926 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:46.887927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:46.887927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:46.887999 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:46.888323 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:46.888324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:46.888325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:46.888446 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:46.888508 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:46.888762 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 14:31:46.888831 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:31:46.889129 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:46.880730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37013 (local address 127.23.173.193:36857)
0812 14:31:46.880901 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:46.880938 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:46.881404 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:46.881606 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:46.881610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:46.882166 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:46.882358 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:46.882363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:46.883014 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:46.883015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:46.883872 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:46.883875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:46.884227 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:46.884227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:46.884344 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:46.884753 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:46.884761 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:46.886111 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:46.887553 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:46.887567 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:46.887570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:46.887855 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:46.888128 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:46.888129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:46.888130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:46.888256 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:46.888506 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:46.888513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:46.888635 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 14:31:46.888745 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 14:31:46.888910 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260812 14:31:46.889627 21087 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37013
I20260812 14:31:46.889716 21087 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:46.889889 21087 heartbeater.cc:511] Master 127.23.173.254:37013 requested a full tablet report, sending...
I20260812 14:31:46.890388 20730 ts_manager.cc:194] Registered new tserver with Master: dc550cd46432430795ee68d1c176ccb4 (127.23.173.193:35877)
I20260812 14:31:46.891251 20730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:36857
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:31:46.912667 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37013
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:46.993091 21092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:46.993307 21092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:46.993341 21092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:46.994583 21092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:46.994630 21092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:46.994665 21092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:46.996202 21092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:46.996457 21092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:46.996665 21092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:47.001421 21106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.001458 21092 server_base.cc:1034] running on GCE node
W20260812 14:31:47.001421 21104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.001397 21109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.001839 21092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:47.002079 21092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:47.003221 21092 hybrid_clock.cc:648] HybridClock initialized: now 1786545107003209 us; error 32 us; skew 500 ppm
Aug 12 14:31:47 dist-test-slave-1l3l krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545107, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:47.005697 21092 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:31:47.006166 21092 webserver.cc:460] Webserver started at http://127.23.173.194:37697/ using document root <none> and password file <none>
I20260812 14:31:47.006358 21092 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:47.006402 21092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:47.006494 21092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:47.007555 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "92d89316af304c089e464a1d2b3d3b15"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "63380e84c33776e345b63dfb51435177"
server_key_iv: "124fbaa068aa2ea0988a7b092b25e59f"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.007901 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "92d89316af304c089e464a1d2b3d3b15"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "63380e84c33776e345b63dfb51435177"
server_key_iv: "124fbaa068aa2ea0988a7b092b25e59f"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.009081 21092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:31:47.009737 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.009895 21092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:47.009972 21092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "92d89316af304c089e464a1d2b3d3b15"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "63380e84c33776e345b63dfb51435177"
server_key_iv: "124fbaa068aa2ea0988a7b092b25e59f"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.010037 21092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:47.019556 21092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:47.021241 21092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:47.021370 21092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:47.021572 21092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:47.021855 21092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:47.021890 21092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.021910 21092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:47.021921 21092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.028509 21092 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:38869
I20260812 14:31:47.028578 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:38869 every 8 connection(s)
I20260812 14:31:47.028916 21092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:31:47 dist-test-slave-1l3l krb5kdc[20643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545107, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:47.037421 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 21092
I20260812 14:31:47.037525 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:31:47.037730 24247 external_mini_cluster.cc:1428] Setting key 491224aee91d5cc96f9c17d17b697b5d
I20260812 14:31:47.038990 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.029959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:55013 (local address 127.23.173.254:37013)
0812 14:31:47.030010 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.030012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.030142 (+ 130us) server_negotiation.cc:407] Connection header received
0812 14:31:47.030797 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.030799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.030812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.030885 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:47.032116 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.032864 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.033937 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.034141 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.034199 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.036299 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:47.036300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:47.036300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:47.036307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:47.037634 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:47.038088 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:47.038089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:47.038090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:47.038154 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:47.038392 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:47.038393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:47.038394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:47.038508 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:47.038568 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.038860 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.038926 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:31:47.039240 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.029669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37013 (local address 127.23.173.194:55013)
0812 14:31:47.029952 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.030064 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.030675 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.030961 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.030965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.031753 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:47.032017 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.032025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.032949 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.032950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.033850 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.033852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.034212 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.034213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.034348 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.034807 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:47.034815 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:47.036215 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:47.037706 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:47.037714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:47.037716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:47.038012 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:47.038212 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:47.038212 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:47.038213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:47.038328 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:47.038568 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:47.038577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:47.038733 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.038838 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.039025 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260812 14:31:47.039741 21326 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37013
I20260812 14:31:47.039842 21326 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:47.040028 21326 heartbeater.cc:511] Master 127.23.173.254:37013 requested a full tablet report, sending...
I20260812 14:31:47.040436 20730 ts_manager.cc:194] Registered new tserver with Master: 92d89316af304c089e464a1d2b3d3b15 (127.23.173.194:38869)
I20260812 14:31:47.040800 20730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:55013
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:31:47.063215 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37013
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:47.144240 21334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:47.144438 21334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:47.144469 21334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:47.145610 21334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:47.145658 21334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:47.145691 21334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:47.147135 21334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:47.147367 21334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:47.147584 21334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:47.152386 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.152400 21351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.152467 21345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.152434 21334 server_base.cc:1034] running on GCE node
I20260812 14:31:47.152765 21334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:47.153024 21334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:47.154199 21334 hybrid_clock.cc:648] HybridClock initialized: now 1786545107154183 us; error 27 us; skew 500 ppm
Aug 12 14:31:47 dist-test-slave-1l3l krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545107, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:47.156728 21334 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:31:47.157157 21334 webserver.cc:460] Webserver started at http://127.23.173.195:36213/ using document root <none> and password file <none>
I20260812 14:31:47.157370 21334 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:47.157418 21334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:47.157533 21334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:47.158536 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "cb55348f5d5a45859a619d98edad20b1"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "8d237ba7ee53b22f5abbdda84d65e4c2"
server_key_iv: "74ea2a5b92f14740a8c4b8c70451b744"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.158946 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "cb55348f5d5a45859a619d98edad20b1"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "8d237ba7ee53b22f5abbdda84d65e4c2"
server_key_iv: "74ea2a5b92f14740a8c4b8c70451b744"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.160239 21334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:47.160905 21359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.161051 21334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:47.161136 21334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "cb55348f5d5a45859a619d98edad20b1"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "8d237ba7ee53b22f5abbdda84d65e4c2"
server_key_iv: "74ea2a5b92f14740a8c4b8c70451b744"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.161199 21334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:47.185472 21334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:47.187234 21334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:47.187366 21334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:47.187580 21334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:47.187896 21334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:47.187932 21334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.187960 21334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:47.187978 21334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.193663 21334 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:36703
I20260812 14:31:47.193723 21537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:36703 every 8 connection(s)
I20260812 14:31:47.194011 21334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:31:47.197989 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 21334
I20260812 14:31:47.198104 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:31:47.198329 24247 external_mini_cluster.cc:1428] Setting key a709518dc47998057091f782674fcee8
Aug 12 14:31:47 dist-test-slave-1l3l krb5kdc[20643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545107, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:47.202610 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.194626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:58235 (local address 127.23.173.254:37013)
0812 14:31:47.194686 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.194688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.194875 (+ 187us) server_negotiation.cc:407] Connection header received
0812 14:31:47.195375 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.195376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.195386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.195430 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:47.196372 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.196866 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.197776 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.197998 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.198067 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.200388 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:47.200389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:47.200389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:47.200395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:47.201630 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:47.201954 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:47.201955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:47.201955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:47.201999 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:47.202170 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:47.202170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:47.202170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:47.202256 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:47.202297 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.202490 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.202542 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:31:47.202767 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.194621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37013 (local address 127.23.173.195:58235)
0812 14:31:47.194774 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.194809 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.195285 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.195494 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.195497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.196095 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:47.196290 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.196295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.196930 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.196930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.197716 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.197718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.198131 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.198132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.198287 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.198888 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:47.198897 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:47.200312 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:47.201693 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:47.201699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:47.201700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:47.201899 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:47.202044 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:47.202044 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:47.202045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:47.202131 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:47.202295 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:47.202298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:47.202399 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.202474 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.202584 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260812 14:31:47.203171 21546 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37013
I20260812 14:31:47.203256 21546 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:47.203420 21546 heartbeater.cc:511] Master 127.23.173.254:37013 requested a full tablet report, sending...
I20260812 14:31:47.203749 20730 ts_manager.cc:194] Registered new tserver with Master: cb55348f5d5a45859a619d98edad20b1 (127.23.173.195:36703)
I20260812 14:31:47.204099 20730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:58235
I20260812 14:31:47.209983 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:47.216691 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.211295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.23.173.254:37013)
0812 14:31:47.211351 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.211353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.211430 (+ 77us) server_negotiation.cc:407] Connection header received
0812 14:31:47.211475 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.211475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.211483 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.211525 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:47.212078 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.212553 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.213212 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.213376 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.213413 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.214401 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:47.214403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:47.214403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:47.214408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:47.215585 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:47.215981 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:47.215983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:47.215983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:47.216048 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:47.216296 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:47.216296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:47.216297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:47.216429 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:47.216501 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.216576 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.216634 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:31:47.218097 20730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:47.219583 20730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:47.225526 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.221641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.23.173.193:35877)
0812 14:31:47.221847 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.221852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.221916 (+ 64us) server_negotiation.cc:407] Connection header received
0812 14:31:47.222018 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.222021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.222036 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.222101 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:47.222244 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.223048 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.224406 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.225120 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.225195 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.225249 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.225397 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.225440 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260812 14:31:47.225544 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.221515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:35877 (local address 127.0.0.1:42698)
0812 14:31:47.221829 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.221868 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.221937 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.222113 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.222116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.222118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:47.222180 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.222182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.223144 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.223145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.224299 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.224303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.225206 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.225207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.225289 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.225302 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.225370 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.225447 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":21,"thread_start_us":56,"threads_started":1}
I20260812 14:31:47.225554 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.221719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:36703 (local address 127.0.0.1:47530)
0812 14:31:47.222266 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.222275 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.222323 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.222583 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.222584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.222586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:47.222636 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.222637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.223799 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.223800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.224636 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.224639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.225319 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.225320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.225383 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.225414 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.225456 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.225487 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260812 14:31:47.225716 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.221800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.23.173.195:36703)
0812 14:31:47.221953 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.221957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.222380 (+ 423us) server_negotiation.cc:407] Connection header received
0812 14:31:47.222432 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.222435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.222449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.222514 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:47.222707 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.223696 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.224708 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.225170 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.225235 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.225288 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.225595 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.225638 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260812 14:31:47.226027 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.221539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:38869 (local address 127.0.0.1:50236)
0812 14:31:47.221843 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.221878 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.221944 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.222184 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.222185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.222188 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:47.222306 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.222308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.223586 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.223587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.224660 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.224662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.225804 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.225805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.225877 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.225881 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.225924 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.225960 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":15,"thread_start_us":56,"threads_started":1}
I20260812 14:31:47.226323 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.221666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.23.173.194:38869)
0812 14:31:47.222042 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.222045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.222054 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:47.222088 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.222090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.222101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.222176 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:47.222390 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.223478 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.224747 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.225439 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.225505 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.225554 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.226199 (+ 645us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.226245 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":88,"threads_started":1}
I20260812 14:31:47.226306 20957 tablet_service.cc:1467] Processing CreateTablet for tablet 6a42ade7f4094a0c99d1eb8d074c907a (DEFAULT_TABLE table=test-table [id=3807e5b8ede243cca5555f3585bc630c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:47.226482 21435 tablet_service.cc:1467] Processing CreateTablet for tablet 6a42ade7f4094a0c99d1eb8d074c907a (DEFAULT_TABLE table=test-table [id=3807e5b8ede243cca5555f3585bc630c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:47.226677 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a42ade7f4094a0c99d1eb8d074c907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:47.226812 21435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a42ade7f4094a0c99d1eb8d074c907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:47.226893 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 6a42ade7f4094a0c99d1eb8d074c907a (DEFAULT_TABLE table=test-table [id=3807e5b8ede243cca5555f3585bc630c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:47.227185 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a42ade7f4094a0c99d1eb8d074c907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:47.229564 21573 tablet_bootstrap.cc:492] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: Bootstrap starting.
I20260812 14:31:47.230194 21573 tablet_bootstrap.cc:654] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:47.230302 21574 tablet_bootstrap.cc:492] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Bootstrap starting.
I20260812 14:31:47.230564 21576 tablet_bootstrap.cc:492] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Bootstrap starting.
I20260812 14:31:47.230970 21574 tablet_bootstrap.cc:654] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:47.230952 21573 log.cc:824] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:47.231123 21576 tablet_bootstrap.cc:654] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:47.231499 21574 log.cc:824] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:47.231683 21576 log.cc:824] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:47.231880 21573 tablet_bootstrap.cc:492] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: No bootstrap required, opened a new log
I20260812 14:31:47.231948 21573 ts_tablet_manager.cc:1397] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:31:47.232306 21574 tablet_bootstrap.cc:492] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: No bootstrap required, opened a new log
I20260812 14:31:47.232386 21574 ts_tablet_manager.cc:1397] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:31:47.232509 21576 tablet_bootstrap.cc:492] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: No bootstrap required, opened a new log
I20260812 14:31:47.232558 21576 ts_tablet_manager.cc:1397] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:31:47.233559 21573 raft_consensus.cc:348] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.233691 21573 raft_consensus.cc:374] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:47.233711 21573 raft_consensus.cc:729] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc550cd46432430795ee68d1c176ccb4, State: Initialized, Role: FOLLOWER
I20260812 14:31:47.233801 21573 consensus_queue.cc:260] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.234072 21087 heartbeater.cc:503] Master 127.23.173.254:37013 was elected leader, sending a full tablet report...
I20260812 14:31:47.234285 21573 ts_tablet_manager.cc:1428] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:47.234269 21574 raft_consensus.cc:348] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.234450 21574 raft_consensus.cc:374] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:47.234483 21574 raft_consensus.cc:729] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb55348f5d5a45859a619d98edad20b1, State: Initialized, Role: FOLLOWER
I20260812 14:31:47.234401 21576 raft_consensus.cc:348] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.234546 21576 raft_consensus.cc:374] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:47.234576 21576 raft_consensus.cc:729] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d89316af304c089e464a1d2b3d3b15, State: Initialized, Role: FOLLOWER
I20260812 14:31:47.234596 21574 consensus_queue.cc:260] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.234670 21576 consensus_queue.cc:260] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.234874 21574 ts_tablet_manager.cc:1428] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:31:47.234915 21576 ts_tablet_manager.cc:1428] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:31:47.234987 21546 heartbeater.cc:503] Master 127.23.173.254:37013 was elected leader, sending a full tablet report...
I20260812 14:31:47.235358 21326 heartbeater.cc:503] Master 127.23.173.254:37013 was elected leader, sending a full tablet report...
W20260812 14:31:47.279636 21327 tablet.cc:2369] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:31:47.354367 21585 raft_consensus.cc:482] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:47.354494 21585 raft_consensus.cc:504] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.354894 21585 leader_election.cc:288] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb55348f5d5a45859a619d98edad20b1 (127.23.173.195:36703), 92d89316af304c089e464a1d2b3d3b15 (127.23.173.194:38869)
I20260812 14:31:47.357730 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.355088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:38869 (local address 127.23.173.193:34705)
0812 14:31:47.355136 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.355148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.355197 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.355424 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.355430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.355433 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:47.355505 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.355507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.356173 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.356174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.356926 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.356928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.357487 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.357488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.357566 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.357569 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.357615 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.357659 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:31:47.357730 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.355191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:34705 (local address 127.23.173.194:38869)
0812 14:31:47.355248 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.355250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.355262 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:31:47.355291 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.355292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.355306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.355406 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:47.355571 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.356100 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.357014 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.357423 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.357489 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.357499 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.357627 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.357662 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:31:47.357885 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.355190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:41207 (local address 127.23.173.195:36703)
0812 14:31:47.355256 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.355258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.355471 (+ 213us) server_negotiation.cc:407] Connection header received
0812 14:31:47.355521 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.355522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.355531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.355581 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:47.355764 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.356263 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.357158 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.357539 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.357602 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.357610 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.357755 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.357797 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:31:47.358032 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.355090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:36703 (local address 127.23.173.193:41207)
0812 14:31:47.355400 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.355410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.355467 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.355594 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.355595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.355596 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:47.355678 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.355685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.356346 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.356347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.357081 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.357083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.357617 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.357618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.357694 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.357697 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.357744 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.357899 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":153,"thread_start_us":61,"threads_started":1}
I20260812 14:31:47.358098 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a42ade7f4094a0c99d1eb8d074c907a" candidate_uuid: "dc550cd46432430795ee68d1c176ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d89316af304c089e464a1d2b3d3b15" is_pre_election: true
I20260812 14:31:47.358278 21237 raft_consensus.cc:2454] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc550cd46432430795ee68d1c176ccb4 in term 0.
I20260812 14:31:47.358361 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a42ade7f4094a0c99d1eb8d074c907a" candidate_uuid: "dc550cd46432430795ee68d1c176ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb55348f5d5a45859a619d98edad20b1" is_pre_election: true
I20260812 14:31:47.358510 21468 raft_consensus.cc:2454] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc550cd46432430795ee68d1c176ccb4 in term 0.
I20260812 14:31:47.358531 20865 leader_election.cc:302] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d89316af304c089e464a1d2b3d3b15, dc550cd46432430795ee68d1c176ccb4; no voters:
I20260812 14:31:47.358649 21585 raft_consensus.cc:2781] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:47.358702 21585 raft_consensus.cc:482] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:47.358731 21585 raft_consensus.cc:3037] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:47.359462 21585 raft_consensus.cc:504] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.359583 21585 leader_election.cc:288] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [CANDIDATE]: Term 1 election: Requested vote from peers cb55348f5d5a45859a619d98edad20b1 (127.23.173.195:36703), 92d89316af304c089e464a1d2b3d3b15 (127.23.173.194:38869)
I20260812 14:31:47.359757 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a42ade7f4094a0c99d1eb8d074c907a" candidate_uuid: "dc550cd46432430795ee68d1c176ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb55348f5d5a45859a619d98edad20b1"
I20260812 14:31:47.359777 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a42ade7f4094a0c99d1eb8d074c907a" candidate_uuid: "dc550cd46432430795ee68d1c176ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d89316af304c089e464a1d2b3d3b15"
I20260812 14:31:47.359846 21237 raft_consensus.cc:3037] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:47.359843 21468 raft_consensus.cc:3037] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:47.360571 21237 raft_consensus.cc:2454] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc550cd46432430795ee68d1c176ccb4 in term 1.
I20260812 14:31:47.360648 21468 raft_consensus.cc:2454] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc550cd46432430795ee68d1c176ccb4 in term 1.
I20260812 14:31:47.360734 20865 leader_election.cc:302] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d89316af304c089e464a1d2b3d3b15, dc550cd46432430795ee68d1c176ccb4; no voters:
I20260812 14:31:47.360821 21585 raft_consensus.cc:2781] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:47.360968 21585 raft_consensus.cc:686] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 1 LEADER]: Becoming Leader. State: Replica: dc550cd46432430795ee68d1c176ccb4, State: Running, Role: LEADER
I20260812 14:31:47.361053 21585 consensus_queue.cc:237] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } }
I20260812 14:31:47.361871 20729 catalog_manager.cc:5495] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc550cd46432430795ee68d1c176ccb4 (127.23.173.193). New cstate: current_term: 1 leader_uuid: "dc550cd46432430795ee68d1c176ccb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc550cd46432430795ee68d1c176ccb4" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 35877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 } health_report { overall_health: UNKNOWN } } }
I20260812 14:31:47.369772 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.367790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.23.173.193:35877)
0812 14:31:47.367835 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.367837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.367846 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:47.367910 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.367911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.367925 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.367993 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:47.368101 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.368533 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.369210 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.369376 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.369416 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.369526 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:47.369627 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:47.369682 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.369698 (+ 16us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.369713 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:31:47.371269 21468 raft_consensus.cc:1261] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 1 FOLLOWER]: Refusing update from remote peer dc550cd46432430795ee68d1c176ccb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:47.371286 21237 raft_consensus.cc:1261] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 1 FOLLOWER]: Refusing update from remote peer dc550cd46432430795ee68d1c176ccb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:47.371572 21595 consensus_queue.cc:1035] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d89316af304c089e464a1d2b3d3b15" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:47.371691 21585 consensus_queue.cc:1035] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb55348f5d5a45859a619d98edad20b1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:47.374845 21605 mvcc.cc:205] Tried to move back new op lower bound from 7317688759790649344 to 7317688759751380992. Current Snapshot: MvccSnapshot[applied={T|T < 7317688759790649344}]
I20260812 14:31:47.376171 20729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:47.376250 20729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:47.376919 20729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd3a3e9f768f473999c086c8f68b6b9f: Sending DeleteTablet for 3 replicas of tablet 6a42ade7f4094a0c99d1eb8d074c907a
I20260812 14:31:47.377242 20957 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a42ade7f4094a0c99d1eb8d074c907a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:47 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:42698
I20260812 14:31:47.377293 21205 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a42ade7f4094a0c99d1eb8d074c907a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:47 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:50236
I20260812 14:31:47.377462 21435 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a42ade7f4094a0c99d1eb8d074c907a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:47 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:47530
I20260812 14:31:47.377483 21621 tablet_replica.cc:330] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: stopping tablet replica
I20260812 14:31:47.377576 21621 raft_consensus.cc:2229] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:31:47.377656 21621 raft_consensus.cc:2258] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:47.377727 21626 tablet_replica.cc:330] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: stopping tablet replica
I20260812 14:31:47.377820 21626 raft_consensus.cc:2229] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:47.377912 21626 raft_consensus.cc:2258] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:47.378008 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 20830
I20260812 14:31:47.378055 21625 tablet_replica.cc:330] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: stopping tablet replica
I20260812 14:31:47.378070 21621 ts_tablet_manager.cc:1905] T 6a42ade7f4094a0c99d1eb8d074c907a P dc550cd46432430795ee68d1c176ccb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:47.378140 21625 raft_consensus.cc:2229] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:47.378224 21625 raft_consensus.cc:2258] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:47.378724 21626 ts_tablet_manager.cc:1905] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:47.381134 21626 ts_tablet_manager.cc:1918] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:47.381213 21626 log.cc:1196] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/6a42ade7f4094a0c99d1eb8d074c907a
I20260812 14:31:47.381448 21626 ts_tablet_manager.cc:1939] T 6a42ade7f4094a0c99d1eb8d074c907a P cb55348f5d5a45859a619d98edad20b1: Deleting consensus metadata
I20260812 14:31:47.381812 20706 catalog_manager.cc:4841] TS cb55348f5d5a45859a619d98edad20b1 (127.23.173.195:36703): tablet 6a42ade7f4094a0c99d1eb8d074c907a (table test-table [id=3807e5b8ede243cca5555f3585bc630c]) successfully deleted
I20260812 14:31:47.383898 21625 ts_tablet_manager.cc:1905] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:47.385512 21625 ts_tablet_manager.cc:1918] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:47.385632 21625 log.cc:1196] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/6a42ade7f4094a0c99d1eb8d074c907a
I20260812 14:31:47.386044 21625 ts_tablet_manager.cc:1939] T 6a42ade7f4094a0c99d1eb8d074c907a P 92d89316af304c089e464a1d2b3d3b15: Deleting consensus metadata
I20260812 14:31:47.386480 20702 catalog_manager.cc:4841] TS 92d89316af304c089e464a1d2b3d3b15 (127.23.173.194:38869): tablet 6a42ade7f4094a0c99d1eb8d074c907a (table test-table [id=3807e5b8ede243cca5555f3585bc630c]) successfully deleted
W20260812 14:31:47.390182 20706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:35877 (error 108)
I20260812 14:31:47.390368 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 21092
I20260812 14:31:47.392727 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.390522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:35877 (local address 127.0.0.1:42718)
0812 14:31:47.392643 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.392692 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2095}
W20260812 14:31:47.392881 20706 catalog_manager.cc:4584] TS dc550cd46432430795ee68d1c176ccb4 (127.23.173.193:35877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a42ade7f4094a0c99d1eb8d074c907a: Network error: Client connection negotiation failed: client connection to 127.23.173.193:35877: connect: Connection refused (error 111)
I20260812 14:31:47.400343 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 21334
I20260812 14:31:47.419040 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 20672
2026-08-12T14:31:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T14:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:47Z Disabled control of system clock
I20260812 14:31:47.453002 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:45149
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:47.536973 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:47.537190 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:47.537210 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:47.538288 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:47.538321 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:47.538332 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:47.538343 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:47.538353 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:47.539616 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:45149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:47.539846 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:47.540107 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:47.544662 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.544685 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.544780 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.545336 21632 server_base.cc:1034] running on GCE node
I20260812 14:31:47.545599 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:47.545864 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:47.547004 21632 hybrid_clock.cc:648] HybridClock initialized: now 1786545107546994 us; error 32 us; skew 500 ppm
I20260812 14:31:47.548245 21632 webserver.cc:460] Webserver started at http://127.23.173.254:46419/ using document root <none> and password file <none>
I20260812 14:31:47.548472 21632 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:47.548522 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:47.548662 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:47.549671 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "82b8083e111e4e31991554242d9276da"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "30a0e81c22ae00f33510e32bf420cc28"
server_key_iv: "2cb2da37fe6b699cad95e439ce626b3f"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.549999 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "82b8083e111e4e31991554242d9276da"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "30a0e81c22ae00f33510e32bf420cc28"
server_key_iv: "2cb2da37fe6b699cad95e439ce626b3f"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.551192 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:31:47.551888 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.552038 21632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:47.552122 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "82b8083e111e4e31991554242d9276da"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "30a0e81c22ae00f33510e32bf420cc28"
server_key_iv: "2cb2da37fe6b699cad95e439ce626b3f"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.552191 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:47.571358 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:47.571686 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:47.571844 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:47.575596 21632 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:45149
I20260812 14:31:47.575634 21747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:45149 every 8 connection(s)
I20260812 14:31:47.575969 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:47.576913 21749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:47.577160 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 21632
I20260812 14:31:47.577247 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:47.577425 24247 external_mini_cluster.cc:1428] Setting key 1a8ac23608842ad91f3ac901de0ae602
I20260812 14:31:47.579972 21749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da: Bootstrap starting.
I20260812 14:31:47.580657 21749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:47.580931 21749 log.cc:824] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:47.581583 21749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da: No bootstrap required, opened a new log
I20260812 14:31:47.583062 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.577929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.23.173.254:45149)
0812 14:31:47.578147 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.578157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.578195 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:31:47.578706 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.578716 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.578799 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.578948 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:47.579106 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.580441 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.581279 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.581685 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.581808 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.582467 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:47.582470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:47.582471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:47.582633 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:47.582666 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:47.582677 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:47.582732 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.582819 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.582900 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260812 14:31:47.583431 21749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8083e111e4e31991554242d9276da" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45149 } }
I20260812 14:31:47.583567 21749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:47.583602 21749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b8083e111e4e31991554242d9276da, State: Initialized, Role: FOLLOWER
I20260812 14:31:47.583743 21749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8083e111e4e31991554242d9276da" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45149 } }
I20260812 14:31:47.583820 21749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:47.583860 21749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:47.583909 21749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:47.584426 21749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8083e111e4e31991554242d9276da" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45149 } }
I20260812 14:31:47.584525 21749 leader_election.cc:302] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82b8083e111e4e31991554242d9276da; no voters:
I20260812 14:31:47.584689 21749 leader_election.cc:288] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:47.584756 21756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:47.584946 21756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [term 1 LEADER]: Becoming Leader. State: Replica: 82b8083e111e4e31991554242d9276da, State: Running, Role: LEADER
I20260812 14:31:47.584955 21749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:47.585059 21756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8083e111e4e31991554242d9276da" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45149 } }
I20260812 14:31:47.585453 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82b8083e111e4e31991554242d9276da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8083e111e4e31991554242d9276da" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45149 } } }
I20260812 14:31:47.585538 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:47.585690 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b8083e111e4e31991554242d9276da. Latest consensus state: current_term: 1 leader_uuid: "82b8083e111e4e31991554242d9276da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8083e111e4e31991554242d9276da" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 45149 } } }
I20260812 14:31:47.585752 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:47.586051 21770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:47.586520 21770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:47.587999 21770 catalog_manager.cc:1347] Generated new cluster ID: 6a63ef65156442ccaec5f635a0803711
I20260812 14:31:47.588053 21770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:47.594777 21770 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:47.595278 21770 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:47.599038 21770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da: Generated new TSK 0
I20260812 14:31:47.599210 21770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:31:47.603585 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45149
--builtin_ntp_servers=127.23.173.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:47.686345 21788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:47.686574 21788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:47.686604 21788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:47.687793 21788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:47.687832 21788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:47.687857 21788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:47.689265 21788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:47.689492 21788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:47.689800 21788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:47.695526 21795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.695668 21788 server_base.cc:1034] running on GCE node
W20260812 14:31:47.695520 21798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.695546 21796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.696002 21788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:47.696271 21788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:47.697432 21788 hybrid_clock.cc:648] HybridClock initialized: now 1786545107697419 us; error 28 us; skew 500 ppm
I20260812 14:31:47.698896 21788 webserver.cc:460] Webserver started at http://127.23.173.193:39515/ using document root <none> and password file <none>
I20260812 14:31:47.699209 21788 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:47.699278 21788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:47.699429 21788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:47.700856 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "2a7ce87467094967b674942de1f0f85f"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "82e0d1dedb67ca1880ab8a923168f3c4"
server_key_iv: "233030e3c97c0033052c97eb344a5b6a"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.701278 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "2a7ce87467094967b674942de1f0f85f"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "82e0d1dedb67ca1880ab8a923168f3c4"
server_key_iv: "233030e3c97c0033052c97eb344a5b6a"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.702978 21788 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260812 14:31:47.703835 21805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.703981 21788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:47.704073 21788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "2a7ce87467094967b674942de1f0f85f"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "82e0d1dedb67ca1880ab8a923168f3c4"
server_key_iv: "233030e3c97c0033052c97eb344a5b6a"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.704138 21788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:47.729621 21788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:47.729935 21788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:47.730079 21788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:47.730316 21788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:47.730649 21788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:47.730684 21788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.730715 21788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:47.730732 21788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.736168 21788 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:37723
I20260812 14:31:47.736238 22008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:37723 every 8 connection(s)
I20260812 14:31:47.736533 21788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:31:47.739266 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 21788
I20260812 14:31:47.739346 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:31:47.739543 24247 external_mini_cluster.cc:1428] Setting key a8cafbf4f14de032aa81a0b81b42d9ee
I20260812 14:31:47.740598 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45149
--builtin_ntp_servers=127.23.173.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:31:47.742115 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.737198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51519 (local address 127.23.173.254:45149)
0812 14:31:47.737262 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.737264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.737516 (+ 252us) server_negotiation.cc:407] Connection header received
0812 14:31:47.738018 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.738020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.738033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.738083 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:47.738777 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.739286 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.740602 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.740878 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.740935 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.741862 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:47.741863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:47.741864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:47.741868 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:47.741885 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:47.741888 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:47.741950 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.742029 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.742056 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:31:47.742367 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.737191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45149 (local address 127.23.173.193:51519)
0812 14:31:47.737383 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.737418 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.737915 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.738161 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.738165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.738173 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:47.738679 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.738684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.739388 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.739389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.740519 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.740523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.740958 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.740959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.741116 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.741709 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:47.741710 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:47.741747 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:47.741775 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:47.741961 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:47.741965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:47.741968 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.742030 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.742147 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260812 14:31:47.742789 22009 heartbeater.cc:347] Connected to a master server at 127.23.173.254:45149
I20260812 14:31:47.742874 22009 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:47.743049 22009 heartbeater.cc:511] Master 127.23.173.254:45149 requested a full tablet report, sending...
I20260812 14:31:47.743517 21687 ts_manager.cc:194] Registered new tserver with Master: 2a7ce87467094967b674942de1f0f85f (127.23.173.193:37723)
I20260812 14:31:47.744179 21687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.193:51519
W20260812 14:31:47.823732 22013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:47.823953 22013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:47.823975 22013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:47.825119 22013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:47.825155 22013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:47.825177 22013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:47.826470 22013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:47.826673 22013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:47.826936 22013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:47.831955 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.831947 22023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.832031 22021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.832448 22013 server_base.cc:1034] running on GCE node
I20260812 14:31:47.832693 22013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:47.832952 22013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:47.834131 22013 hybrid_clock.cc:648] HybridClock initialized: now 1786545107834112 us; error 31 us; skew 500 ppm
I20260812 14:31:47.835517 22013 webserver.cc:460] Webserver started at http://127.23.173.194:40331/ using document root <none> and password file <none>
I20260812 14:31:47.835804 22013 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:47.835855 22013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:47.835988 22013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:47.837109 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "ceb1bbb9328d4edb94e884e68ceb582e"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "c0e20ec635a3b91d2e5dbde863d1371a"
server_key_iv: "dec5fa47f921c1eca7e1528f1d4a5ce9"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.837463 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "ceb1bbb9328d4edb94e884e68ceb582e"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "c0e20ec635a3b91d2e5dbde863d1371a"
server_key_iv: "dec5fa47f921c1eca7e1528f1d4a5ce9"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.838774 22013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:31:47.839689 22034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.839939 22013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:47.840027 22013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "ceb1bbb9328d4edb94e884e68ceb582e"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "c0e20ec635a3b91d2e5dbde863d1371a"
server_key_iv: "dec5fa47f921c1eca7e1528f1d4a5ce9"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.840106 22013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:47.866770 22013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:47.867123 22013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:47.867268 22013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:47.867524 22013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:47.867940 22013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:47.867980 22013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.868011 22013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:47.868036 22013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.873699 22013 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:34637
I20260812 14:31:47.873762 22224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:34637 every 8 connection(s)
I20260812 14:31:47.874105 22013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:31:47.877753 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 22013
I20260812 14:31:47.877873 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:31:47.878073 24247 external_mini_cluster.cc:1428] Setting key eac824ec1f899337047797c249fb1d30
I20260812 14:31:47.879055 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45149
--builtin_ntp_servers=127.23.173.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:31:47.879825 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.874900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:56345 (local address 127.23.173.254:45149)
0812 14:31:47.874992 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.874994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.875184 (+ 190us) server_negotiation.cc:407] Connection header received
0812 14:31:47.875806 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.875807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.875820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.875875 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:47.876580 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.877090 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.878352 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.878611 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.878674 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.879537 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:47.879539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:47.879539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:47.879544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:47.879561 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:47.879565 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:47.879613 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:47.879691 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 14:31:47.879748 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 14:31:47.880031 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.874836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45149 (local address 127.23.173.194:56345)
0812 14:31:47.875055 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.875094 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.875662 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.875953 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.875957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.875966 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:47.876485 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.876490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.877187 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.877188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.878267 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:47.878271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:47.878715 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:47.878717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:47.878849 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:47.879394 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:47.879394 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:47.879425 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:47.879450 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:47.879620 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:47.879626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:47.879629 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:47.879690 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:31:47.879823 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260812 14:31:47.880481 22226 heartbeater.cc:347] Connected to a master server at 127.23.173.254:45149
I20260812 14:31:47.880582 22226 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:47.880797 22226 heartbeater.cc:511] Master 127.23.173.254:45149 requested a full tablet report, sending...
I20260812 14:31:47.881176 21687 ts_manager.cc:194] Registered new tserver with Master: ceb1bbb9328d4edb94e884e68ceb582e (127.23.173.194:34637)
I20260812 14:31:47.881554 21687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.194:56345
W20260812 14:31:47.963107 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:47.963338 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:47.963370 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:47.964581 22232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:47.964634 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:47.964670 22232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:47.966027 22232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:47.966249 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:47.966521 22232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:47.971644 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.971662 22232 server_base.cc:1034] running on GCE node
W20260812 14:31:47.971680 22243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:47.971668 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:47.972082 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:47.972306 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:47.973431 22232 hybrid_clock.cc:648] HybridClock initialized: now 1786545107973421 us; error 27 us; skew 500 ppm
I20260812 14:31:47.974642 22232 webserver.cc:460] Webserver started at http://127.23.173.195:36185/ using document root <none> and password file <none>
I20260812 14:31:47.974881 22232 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:47.974927 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:47.975044 22232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:47.976166 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "73fbafb7fe7b44729eae0ebbd65718ed"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "aa396e7a10342dbc947056806672bffc"
server_key_iv: "168fd4139188bd8b8dcea8b922f21baf"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.976472 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "73fbafb7fe7b44729eae0ebbd65718ed"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "aa396e7a10342dbc947056806672bffc"
server_key_iv: "168fd4139188bd8b8dcea8b922f21baf"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.977658 22232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:47.978294 22251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.978423 22232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.978500 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "73fbafb7fe7b44729eae0ebbd65718ed"
format_stamp: "Formatted at 2026-08-12 14:31:47 on dist-test-slave-1l3l"
server_key: "aa396e7a10342dbc947056806672bffc"
server_key_iv: "168fd4139188bd8b8dcea8b922f21baf"
server_key_version: "encryptionkey@0"
I20260812 14:31:47.978571 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:47.990571 22232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:47.990898 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:47.991048 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:47.991289 22232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:47.991639 22232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:47.991674 22232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.991726 22232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:47.991753 22232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:47.997570 22232 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:34121
I20260812 14:31:47.997644 22465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:34121 every 8 connection(s)
I20260812 14:31:47.997958 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:31:48.003289 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.998754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:34863 (local address 127.23.173.254:45149)
0812 14:31:47.998822 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:47.998824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:47.999063 (+ 239us) server_negotiation.cc:407] Connection header received
0812 14:31:47.999606 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:47.999607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:47.999620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:47.999675 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:48.000447 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.000943 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.001915 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.002113 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.002170 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.003016 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:48.003017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:48.003017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:48.003023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:48.003038 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:48.003043 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:48.003088 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.003173 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.003204 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:31:48.003484 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:47.998741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:45149 (local address 127.23.173.195:34863)
0812 14:31:47.998933 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:47.998970 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:47.999491 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:47.999777 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:47.999781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:47.999789 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:48.000339 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.000344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.001037 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.001038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.001815 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.001817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.002190 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.002191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.002317 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.002861 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:48.002861 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:48.002900 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:48.002923 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:48.003094 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:48.003100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:48.003104 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:48.003168 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:31:48.003277 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260812 14:31:48.003988 22468 heartbeater.cc:347] Connected to a master server at 127.23.173.254:45149
I20260812 14:31:48.004082 22468 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:48.004264 22468 heartbeater.cc:511] Master 127.23.173.254:45149 requested a full tablet report, sending...
I20260812 14:31:48.004643 21687 ts_manager.cc:194] Registered new tserver with Master: 73fbafb7fe7b44729eae0ebbd65718ed (127.23.173.195:34121)
I20260812 14:31:48.004997 21687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.195:34863
I20260812 14:31:48.005328 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 22232
I20260812 14:31:48.005429 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:31:48.005642 24247 external_mini_cluster.cc:1428] Setting key 801344503a1e0796be5a7caa4c5895d6
I20260812 14:31:48.006479 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:48.010950 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.007890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.23.173.254:45149)
0812 14:31:48.007950 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.007952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.008081 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:31:48.008136 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.008136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.008147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.008200 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:48.008356 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.008828 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.009779 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.009988 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.010046 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.010708 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:48.010709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:48.010710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:48.010715 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:48.010731 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:48.010736 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:48.010781 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.010834 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.010867 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:31:48.012645 21687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:48.014213 21687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:48.020274 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.016431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:37723 (local address 127.0.0.1:55280)
0812 14:31:48.016585 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:48.016631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:48.016704 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:48.016933 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:48.016937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:48.016944 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:48.017020 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.017023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.018087 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.018088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.018985 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.018987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.019945 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.019946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.020033 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.020049 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 14:31:48.020133 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:31:48.020188 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":216,"threads_started":1}
I20260812 14:31:48.020292 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.016585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.23.173.193:37723)
0812 14:31:48.016741 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.016746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.016757 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:48.016804 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.016806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.016823 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.016924 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:48.017098 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.018011 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.019081 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.019855 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.019944 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.020048 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.020155 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.020199 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260812 14:31:48.020941 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.016457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:34121 (local address 127.0.0.1:34494)
0812 14:31:48.016901 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:48.016909 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:48.016943 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:48.017426 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:48.017427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:48.017430 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:48.017475 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.017476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.018791 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.018792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.019685 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.019688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.020702 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.020703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.020770 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.020773 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:48.020822 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:31:48.020871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":21,"thread_start_us":207,"threads_started":1}
I20260812 14:31:48.020970 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.017093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.23.173.195:34121)
0812 14:31:48.017270 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.017274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.017284 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:31:48.017320 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.017322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.017336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.017416 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:48.017553 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.018704 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.019803 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.020501 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.020576 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.020666 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.020851 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.020895 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260812 14:31:48.021293 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.016431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:34637 (local address 127.0.0.1:36380)
0812 14:31:48.016768 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:48.016778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:48.016826 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:48.017493 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:48.017494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:48.017496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:48.017540 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.017541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.018781 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.018782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.019934 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.019937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.020976 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.020977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.021151 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.021156 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:31:48.021187 (+ 31us) client_negotiation.cc:240] Negotiation successful
0812 14:31:48.021226 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":42,"thread_start_us":78,"threads_started":1}
I20260812 14:31:48.021180 21894 tablet_service.cc:1467] Processing CreateTablet for tablet 5f17eec762a9456b8ab60214eeb855c3 (DEFAULT_TABLE table=test-table [id=b63dc64575e2426ebb66b04e93f61ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:48.021596 21894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f17eec762a9456b8ab60214eeb855c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:48.021708 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.016827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.23.173.194:34637)
0812 14:31:48.017310 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.017315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.017325 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:31:48.017363 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.017365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.017380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.017483 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:48.017599 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.018686 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.020023 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.020880 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.021475 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.021567 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.021592 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.021632 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":153,"threads_started":1}
I20260812 14:31:48.021631 22347 tablet_service.cc:1467] Processing CreateTablet for tablet 5f17eec762a9456b8ab60214eeb855c3 (DEFAULT_TABLE table=test-table [id=b63dc64575e2426ebb66b04e93f61ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:48.021975 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f17eec762a9456b8ab60214eeb855c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:48.022337 22116 tablet_service.cc:1467] Processing CreateTablet for tablet 5f17eec762a9456b8ab60214eeb855c3 (DEFAULT_TABLE table=test-table [id=b63dc64575e2426ebb66b04e93f61ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:48.022737 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f17eec762a9456b8ab60214eeb855c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:48.024489 22496 tablet_bootstrap.cc:492] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: Bootstrap starting.
I20260812 14:31:48.025158 22496 tablet_bootstrap.cc:654] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:48.025491 22496 log.cc:824] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:48.025502 22498 tablet_bootstrap.cc:492] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Bootstrap starting.
I20260812 14:31:48.026219 22498 tablet_bootstrap.cc:654] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:48.026230 22501 tablet_bootstrap.cc:492] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Bootstrap starting.
I20260812 14:31:48.026551 22498 log.cc:824] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:48.026855 22501 tablet_bootstrap.cc:654] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:48.027179 22501 log.cc:824] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:48.027256 22498 tablet_bootstrap.cc:492] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: No bootstrap required, opened a new log
I20260812 14:31:48.027323 22498 ts_tablet_manager.cc:1397] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:48.027323 22496 tablet_bootstrap.cc:492] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: No bootstrap required, opened a new log
I20260812 14:31:48.027388 22496 ts_tablet_manager.cc:1397] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 14:31:48.028065 22501 tablet_bootstrap.cc:492] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: No bootstrap required, opened a new log
I20260812 14:31:48.028139 22501 ts_tablet_manager.cc:1397] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:48.029451 22498 raft_consensus.cc:348] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.029590 22498 raft_consensus.cc:374] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:48.029623 22498 raft_consensus.cc:729] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fbafb7fe7b44729eae0ebbd65718ed, State: Initialized, Role: FOLLOWER
I20260812 14:31:48.029737 22498 consensus_queue.cc:260] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.029848 22496 raft_consensus.cc:348] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.030026 22496 raft_consensus.cc:374] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:48.030028 22498 ts_tablet_manager.cc:1428] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:31:48.030066 22496 raft_consensus.cc:729] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7ce87467094967b674942de1f0f85f, State: Initialized, Role: FOLLOWER
I20260812 14:31:48.030153 22468 heartbeater.cc:503] Master 127.23.173.254:45149 was elected leader, sending a full tablet report...
I20260812 14:31:48.030246 22496 consensus_queue.cc:260] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.030514 22496 ts_tablet_manager.cc:1428] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:48.030573 22009 heartbeater.cc:503] Master 127.23.173.254:45149 was elected leader, sending a full tablet report...
I20260812 14:31:48.030721 22501 raft_consensus.cc:348] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.030839 22501 raft_consensus.cc:374] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:48.030872 22501 raft_consensus.cc:729] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb1bbb9328d4edb94e884e68ceb582e, State: Initialized, Role: FOLLOWER
I20260812 14:31:48.030978 22501 consensus_queue.cc:260] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.031235 22501 ts_tablet_manager.cc:1428] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:48.031278 22226 heartbeater.cc:503] Master 127.23.173.254:45149 was elected leader, sending a full tablet report...
W20260812 14:31:48.124688 22227 tablet.cc:2369] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:31:48.237227 22011 tablet.cc:2369] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:31:48.248538 22469 tablet.cc:2369] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:31:48.339999 22510 raft_consensus.cc:482] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:48.340144 22510 raft_consensus.cc:504] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.340533 22510 leader_election.cc:288] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7ce87467094967b674942de1f0f85f (127.23.173.193:37723), 73fbafb7fe7b44729eae0ebbd65718ed (127.23.173.195:34121)
I20260812 14:31:48.343544 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.340792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:44363 (local address 127.23.173.193:37723)
0812 14:31:48.340841 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.340844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.341004 (+ 160us) server_negotiation.cc:407] Connection header received
0812 14:31:48.341071 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.341072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.341086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.341135 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:48.341305 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.341789 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.342837 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.343206 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.343262 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.343270 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.343427 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.343469 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:31:48.343568 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.340725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:34121 (local address 127.23.173.194:56455)
0812 14:31:48.340797 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:48.340809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:48.340856 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:48.341042 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:48.341043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:48.341045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:48.341129 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.341131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.341787 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.341789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.342739 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.342742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.343307 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.343308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.343390 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.343393 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:48.343447 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:31:48.343484 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":5}
I20260812 14:31:48.343572 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.340835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:56455 (local address 127.23.173.195:34121)
0812 14:31:48.340913 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.340915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.340926 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:48.340962 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.340963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.340972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.341030 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:48.341203 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.341711 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.342826 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.343229 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.343300 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.343308 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.343463 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.343505 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:31:48.343588 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.340727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:37723 (local address 127.23.173.194:44363)
0812 14:31:48.340923 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:48.340933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:48.341002 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:48.341145 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:48.341145 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:48.341147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:48.341228 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.341230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.341887 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.341888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.342738 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:48.342741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.343289 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.343290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:48.343363 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.343366 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:48.343408 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:31:48.343446 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":28,"thread_start_us":72,"threads_started":1}
I20260812 14:31:48.343940 22380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f17eec762a9456b8ab60214eeb855c3" candidate_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" is_pre_election: true
I20260812 14:31:48.344125 22380 raft_consensus.cc:2454] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb1bbb9328d4edb94e884e68ceb582e in term 0.
I20260812 14:31:48.344153 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f17eec762a9456b8ab60214eeb855c3" candidate_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7ce87467094967b674942de1f0f85f" is_pre_election: true
I20260812 14:31:48.344314 21929 raft_consensus.cc:2454] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb1bbb9328d4edb94e884e68ceb582e in term 0.
I20260812 14:31:48.344556 22036 leader_election.cc:302] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7ce87467094967b674942de1f0f85f, ceb1bbb9328d4edb94e884e68ceb582e; no voters:
I20260812 14:31:48.344694 22510 raft_consensus.cc:2781] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:48.344743 22510 raft_consensus.cc:482] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:48.344771 22510 raft_consensus.cc:3037] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:48.345504 22510 raft_consensus.cc:504] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.345630 22510 leader_election.cc:288] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [CANDIDATE]: Term 1 election: Requested vote from peers 2a7ce87467094967b674942de1f0f85f (127.23.173.193:37723), 73fbafb7fe7b44729eae0ebbd65718ed (127.23.173.195:34121)
I20260812 14:31:48.345784 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f17eec762a9456b8ab60214eeb855c3" candidate_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7ce87467094967b674942de1f0f85f"
I20260812 14:31:48.345873 21929 raft_consensus.cc:3037] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:48.346585 21929 raft_consensus.cc:2454] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb1bbb9328d4edb94e884e68ceb582e in term 1.
I20260812 14:31:48.346685 22380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f17eec762a9456b8ab60214eeb855c3" candidate_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fbafb7fe7b44729eae0ebbd65718ed"
I20260812 14:31:48.346750 22036 leader_election.cc:302] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7ce87467094967b674942de1f0f85f, ceb1bbb9328d4edb94e884e68ceb582e; no voters:
I20260812 14:31:48.346762 22380 raft_consensus.cc:3037] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:48.346837 22510 raft_consensus.cc:2781] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:48.346993 22510 raft_consensus.cc:686] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 1 LEADER]: Becoming Leader. State: Replica: ceb1bbb9328d4edb94e884e68ceb582e, State: Running, Role: LEADER
I20260812 14:31:48.347090 22510 consensus_queue.cc:237] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } }
I20260812 14:31:48.347661 22380 raft_consensus.cc:2454] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb1bbb9328d4edb94e884e68ceb582e in term 1.
I20260812 14:31:48.348214 21687 catalog_manager.cc:5495] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb1bbb9328d4edb94e884e68ceb582e (127.23.173.194). New cstate: current_term: 1 leader_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb1bbb9328d4edb94e884e68ceb582e" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 34637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 } health_report { overall_health: UNKNOWN } } }
I20260812 14:31:48.376049 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.373843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.23.173.194:34637)
0812 14:31:48.373894 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:48.373896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:48.373905 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:48.373937 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:48.373938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:48.373955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:48.374012 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:48.374112 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.374563 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.375323 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:48.375511 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:48.375571 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:48.375768 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:48.375869 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:48.375931 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:48.375961 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 14:31:48.375976 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:31:48.377714 22380 raft_consensus.cc:1261] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 1 FOLLOWER]: Refusing update from remote peer ceb1bbb9328d4edb94e884e68ceb582e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:48.377722 21929 raft_consensus.cc:1261] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 1 FOLLOWER]: Refusing update from remote peer ceb1bbb9328d4edb94e884e68ceb582e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:48.378039 22514 consensus_queue.cc:1035] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7ce87467094967b674942de1f0f85f" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 37723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:48.378165 22510 consensus_queue.cc:1035] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [LEADER]: Connected to new peer: Peer: permanent_uuid: "73fbafb7fe7b44729eae0ebbd65718ed" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:48.380947 22517 mvcc.cc:205] Tried to move back new op lower bound from 7317688763912761344 to 7317688763790098432. Current Snapshot: MvccSnapshot[applied={T|T < 7317688763912761344 or (T in {7317688763912761344})}]
I20260812 14:31:48.381502 22520 mvcc.cc:205] Tried to move back new op lower bound from 7317688763912761344 to 7317688763790098432. Current Snapshot: MvccSnapshot[applied={T|T < 7317688763912761344}]
I20260812 14:31:48.382619 21687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:48.382714 21687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:48.383493 21687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b8083e111e4e31991554242d9276da: Sending DeleteTablet for 3 replicas of tablet 5f17eec762a9456b8ab60214eeb855c3
I20260812 14:31:48.383883 22347 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f17eec762a9456b8ab60214eeb855c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:48 UTC) from {username='slave'} at 127.0.0.1:34494
I20260812 14:31:48.383915 22116 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f17eec762a9456b8ab60214eeb855c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:48 UTC) from {username='slave'} at 127.0.0.1:36380
I20260812 14:31:48.383957 21894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f17eec762a9456b8ab60214eeb855c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:48 UTC) from {username='slave'} at 127.0.0.1:55280
I20260812 14:31:48.384179 22534 tablet_replica.cc:330] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f: stopping tablet replica
I20260812 14:31:48.384284 22534 raft_consensus.cc:2229] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:48.384341 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 21788
I20260812 14:31:48.384414 22534 raft_consensus.cc:2258] T 5f17eec762a9456b8ab60214eeb855c3 P 2a7ce87467094967b674942de1f0f85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:48.394845 22533 tablet_replica.cc:330] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: stopping tablet replica
I20260812 14:31:48.395031 22533 raft_consensus.cc:2229] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:31:48.395215 22533 raft_consensus.cc:2258] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:48.395797 22533 ts_tablet_manager.cc:1905] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:48.397481 22533 ts_tablet_manager.cc:1918] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:48.397603 22533 log.cc:1196] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/5f17eec762a9456b8ab60214eeb855c3
I20260812 14:31:48.397881 22533 ts_tablet_manager.cc:1939] T 5f17eec762a9456b8ab60214eeb855c3 P ceb1bbb9328d4edb94e884e68ceb582e: Deleting consensus metadata
I20260812 14:31:48.398307 22531 tablet_replica.cc:330] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: stopping tablet replica
I20260812 14:31:48.398350 21664 catalog_manager.cc:4841] TS ceb1bbb9328d4edb94e884e68ceb582e (127.23.173.194:34637): tablet 5f17eec762a9456b8ab60214eeb855c3 (table test-table [id=b63dc64575e2426ebb66b04e93f61ac4]) successfully deleted
I20260812 14:31:48.398464 22531 raft_consensus.cc:2229] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:48.398595 22531 raft_consensus.cc:2258] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:48.399080 22531 ts_tablet_manager.cc:1905] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:31:48.399263 21659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:37723 (error 108)
I20260812 14:31:48.399734 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:48.399583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:37723 (local address 127.0.0.1:55234)
0812 14:31:48.399634 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:48.399679 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:31:48.399842 21659 catalog_manager.cc:4584] TS 2a7ce87467094967b674942de1f0f85f (127.23.173.193:37723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f17eec762a9456b8ab60214eeb855c3: Network error: Client connection negotiation failed: client connection to 127.23.173.193:37723: connect: Connection refused (error 111)
I20260812 14:31:48.400123 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 22013
I20260812 14:31:48.412796 22531 ts_tablet_manager.cc:1918] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:48.412883 22531 log.cc:1196] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/5f17eec762a9456b8ab60214eeb855c3
I20260812 14:31:48.413103 22531 ts_tablet_manager.cc:1939] T 5f17eec762a9456b8ab60214eeb855c3 P 73fbafb7fe7b44729eae0ebbd65718ed: Deleting consensus metadata
I20260812 14:31:48.414288 21665 catalog_manager.cc:4841] TS 73fbafb7fe7b44729eae0ebbd65718ed (127.23.173.195:34121): tablet 5f17eec762a9456b8ab60214eeb855c3 (table test-table [id=b63dc64575e2426ebb66b04e93f61ac4]) successfully deleted
I20260812 14:31:48.419371 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 22232
I20260812 14:31:48.430076 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 21632
2026-08-12T14:31:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:48 dist-test-slave-1l3l krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:48 dist-test-slave-1l3l krb5kdc[22543](info): set up 2 sockets
Aug 12 14:31:48 dist-test-slave-1l3l krb5kdc[22543](info): commencing operation
krb5kdc: starting...
W20260812 14:31:49.615554 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:49 dist-test-slave-1l3l krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:31:49.736783 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37227
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:49.816607 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:49.816794 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:49.816814 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:49.817855 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:49.817886 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:49.817899 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:49.817910 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:49.817921 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:49.819603 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:49.819847 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:49.820034 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:49.824602 22576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:49.824715 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:49.824817 22563 server_base.cc:1034] running on GCE node
W20260812 14:31:49.824869 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:49.825133 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:49.825376 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:49.826521 22563 hybrid_clock.cc:648] HybridClock initialized: now 1786545109826498 us; error 35 us; skew 500 ppm
Aug 12 14:31:49 dist-test-slave-1l3l krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545109, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:49.828938 22563 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:31:49.829344 22563 webserver.cc:460] Webserver started at http://127.23.173.254:35387/ using document root <none> and password file <none>
I20260812 14:31:49.829561 22563 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:49.829610 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:49.829736 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:49.830757 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "201433eadd974c468516ebe9ce785233"
format_stamp: "Formatted at 2026-08-12 14:31:49 on dist-test-slave-1l3l"
server_key: "1bd668c43b1a182228479c56418cd368"
server_key_iv: "f8b438b7810beb9023f9a5d4696a7552"
server_key_version: "encryptionkey@0"
I20260812 14:31:49.831094 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "201433eadd974c468516ebe9ce785233"
format_stamp: "Formatted at 2026-08-12 14:31:49 on dist-test-slave-1l3l"
server_key: "1bd668c43b1a182228479c56418cd368"
server_key_iv: "f8b438b7810beb9023f9a5d4696a7552"
server_key_version: "encryptionkey@0"
I20260812 14:31:49.832507 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:49.833170 22586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:49.833317 22563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:49.833400 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "201433eadd974c468516ebe9ce785233"
format_stamp: "Formatted at 2026-08-12 14:31:49 on dist-test-slave-1l3l"
server_key: "1bd668c43b1a182228479c56418cd368"
server_key_iv: "f8b438b7810beb9023f9a5d4696a7552"
server_key_version: "encryptionkey@0"
I20260812 14:31:49.833459 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:49.846310 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:49.848048 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:49.848183 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:49.851976 22563 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:37227
I20260812 14:31:49.852027 22683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:37227 every 8 connection(s)
I20260812 14:31:49.852331 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:49.853137 22684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:49.855255 22684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233: Bootstrap starting.
I20260812 14:31:49.855827 22684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:49.856062 22684 log.cc:824] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:49.856873 22684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233: No bootstrap required, opened a new log
I20260812 14:31:49.858969 22684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201433eadd974c468516ebe9ce785233" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37227 } }
I20260812 14:31:49.859153 22684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:49.859189 22684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201433eadd974c468516ebe9ce785233, State: Initialized, Role: FOLLOWER
I20260812 14:31:49.859308 22684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201433eadd974c468516ebe9ce785233" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37227 } }
I20260812 14:31:49.859360 22684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:49.859386 22684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:49.859417 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:49.859964 22684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201433eadd974c468516ebe9ce785233" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37227 } }
I20260812 14:31:49.860052 22684 leader_election.cc:302] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201433eadd974c468516ebe9ce785233; no voters:
I20260812 14:31:49.860193 22684 leader_election.cc:288] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:49.860231 22688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:49.860396 22688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [term 1 LEADER]: Becoming Leader. State: Replica: 201433eadd974c468516ebe9ce785233, State: Running, Role: LEADER
I20260812 14:31:49.860402 22684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:49.860489 22688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201433eadd974c468516ebe9ce785233" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37227 } }
I20260812 14:31:49.860850 22691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201433eadd974c468516ebe9ce785233. Latest consensus state: current_term: 1 leader_uuid: "201433eadd974c468516ebe9ce785233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201433eadd974c468516ebe9ce785233" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37227 } } }
I20260812 14:31:49.860934 22691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:49.861068 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "201433eadd974c468516ebe9ce785233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201433eadd974c468516ebe9ce785233" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37227 } } }
I20260812 14:31:49.861135 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:49.861366 22705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:49.861686 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 22563
I20260812 14:31:49.861802 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:49.861877 22705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:49.861995 24247 external_mini_cluster.cc:1428] Setting key 31fc42ee11303208026db67c6ba6f942
I20260812 14:31:49.863409 22705 catalog_manager.cc:1347] Generated new cluster ID: 445fbb239e7647b4adb62de53e8f2c4a
I20260812 14:31:49.863462 22705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:31:49 dist-test-slave-1l3l krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:49.877342 22705 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:49.877607 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:49.862539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.23.173.254:37227)
0812 14:31:49.863049 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:49.863058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:49.863094 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:31:49.863205 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:49.863212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:49.863264 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:49.867803 (+ 4539us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:49.867907 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:49.869310 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:49.870433 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:49.870997 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:49.871099 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:49.873853 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:49.873858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:49.873858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:49.873871 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:49.875839 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:49.876235 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:49.876239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:49.876239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:49.876349 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:49.876532 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:49.876533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:49.876533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:49.876843 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:49.877019 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:49.877125 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 14:31:49.877266 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":71,"threads_started":1}
I20260812 14:31:49.877913 22705 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:49.885031 22705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233: Generated new TSK 0
I20260812 14:31:49.885191 22705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:31:49.914119 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37227
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:49.995723 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:49.995918 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:49.995940 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:49.997012 22728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:49.997044 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:49.997067 22728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:49.998595 22728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:49.998780 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:49.998953 22728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:50.003849 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:50.003881 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.004040 22728 server_base.cc:1034] running on GCE node
W20260812 14:31:50.003887 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.004272 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:50.004516 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:50.005666 22728 hybrid_clock.cc:648] HybridClock initialized: now 1786545110005653 us; error 30 us; skew 500 ppm
Aug 12 14:31:50 dist-test-slave-1l3l krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545110, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:50.008114 22728 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:31:50.008570 22728 webserver.cc:460] Webserver started at http://127.23.173.193:37905/ using document root <none> and password file <none>
I20260812 14:31:50.008770 22728 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:50.008821 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:50.008940 22728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:50.009935 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "35e3c8fa151f4066b3bc0a79d8e86636"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "601343e8209178a6fa461b87d40b46d9"
server_key_iv: "d92bc2b4d9cf6dd8ed665b17eb66b0c9"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.010267 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "35e3c8fa151f4066b3bc0a79d8e86636"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "601343e8209178a6fa461b87d40b46d9"
server_key_iv: "d92bc2b4d9cf6dd8ed665b17eb66b0c9"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.011487 22728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:31:50.012219 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.012365 22728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.012444 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "35e3c8fa151f4066b3bc0a79d8e86636"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "601343e8209178a6fa461b87d40b46d9"
server_key_iv: "d92bc2b4d9cf6dd8ed665b17eb66b0c9"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.012511 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:50.023379 22728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:50.025421 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:50.025549 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:50.025754 22728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:50.026026 22728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:50.026059 22728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.026086 22728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:50.026103 22728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.031592 22728 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:34969
I20260812 14:31:50.031649 22950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:34969 every 8 connection(s)
I20260812 14:31:50.031935 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:31:50 dist-test-slave-1l3l krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545110, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:50.038836 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 22728
I20260812 14:31:50.038936 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:31:50.039117 24247 external_mini_cluster.cc:1428] Setting key 4a3969c20abb528cd06c31adfe216cf3
I20260812 14:31:50.040527 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.032546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:43819 (local address 127.23.173.254:37227)
0812 14:31:50.032589 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.032591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.032757 (+ 166us) server_negotiation.cc:407] Connection header received
0812 14:31:50.033243 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.033244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.033258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.033314 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:50.034214 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.034744 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.035763 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.035977 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.036034 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.037943 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:50.037944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:50.037944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:50.037952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:50.039113 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.039528 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.039529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.039529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.039597 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.039889 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.039891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.039891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.040009 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:50.040068 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.040325 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.040432 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:31:50.040638 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.032508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37227 (local address 127.23.173.193:43819)
0812 14:31:50.032641 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.032682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.033146 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.033392 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.033396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.033937 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:50.034114 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.034119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.034832 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.034833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.035643 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.035646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.036051 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.036052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.036167 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.036564 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:50.036572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:50.037855 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:50.039178 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:50.039186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:50.039188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:50.039466 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:50.039656 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:50.039657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:50.039658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:50.039813 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:50.040068 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:50.040075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:50.040202 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.040291 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.040438 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260812 14:31:50.041081 22953 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37227
I20260812 14:31:50.041181 22953 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:50.041355 22953 heartbeater.cc:511] Master 127.23.173.254:37227 requested a full tablet report, sending...
I20260812 14:31:50.041872 22619 ts_manager.cc:194] Registered new tserver with Master: 35e3c8fa151f4066b3bc0a79d8e86636 (127.23.173.193:34969)
I20260812 14:31:50.042438 22619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:43819
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:31:50.066910 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37227
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:50.150636 22963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:50.150849 22963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:50.150873 22963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:50.152038 22963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:50.152081 22963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:50.152107 22963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:50.153529 22963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:50.153723 22963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:50.153914 22963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:50.158885 22976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:50.158962 22984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.159070 22963 server_base.cc:1034] running on GCE node
W20260812 14:31:50.159155 22972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.159348 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:50.159590 22963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:50.160733 22963 hybrid_clock.cc:648] HybridClock initialized: now 1786545110160717 us; error 33 us; skew 500 ppm
Aug 12 14:31:50 dist-test-slave-1l3l krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545110, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:50.163365 22963 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:31:50.163875 22963 webserver.cc:460] Webserver started at http://127.23.173.194:40497/ using document root <none> and password file <none>
I20260812 14:31:50.164095 22963 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:50.164145 22963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:50.164275 22963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:50.165319 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "09d3412cef1b4e0684752e61f68de459"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "fc3863bfa3c2fa7eef6b5d211402c6d5"
server_key_iv: "323e6f1f135fc44722b4ca0309d9c008"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.165658 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "09d3412cef1b4e0684752e61f68de459"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "fc3863bfa3c2fa7eef6b5d211402c6d5"
server_key_iv: "323e6f1f135fc44722b4ca0309d9c008"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.166958 22963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:31:50.167675 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.167830 22963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:50.167915 22963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "09d3412cef1b4e0684752e61f68de459"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "fc3863bfa3c2fa7eef6b5d211402c6d5"
server_key_iv: "323e6f1f135fc44722b4ca0309d9c008"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.167984 22963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:50.190002 22963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:50.192247 22963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:50.192400 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:50.192641 22963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:50.193084 22963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:50.193156 22963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.193197 22963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:50.193214 22963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.198983 22963 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:38381
I20260812 14:31:50.199044 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:38381 every 8 connection(s)
I20260812 14:31:50.199337 22963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:31:50.202668 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 22963
I20260812 14:31:50.202764 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:31:50.202940 24247 external_mini_cluster.cc:1428] Setting key d612499589e8d054c541770b3e28ecff
Aug 12 14:31:50 dist-test-slave-1l3l krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545110, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:50.208559 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.200018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:36787 (local address 127.23.173.254:37227)
0812 14:31:50.200063 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.200065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.200307 (+ 242us) server_negotiation.cc:407] Connection header received
0812 14:31:50.200829 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.200830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.200841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.200887 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:50.201789 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.202278 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.203307 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.203522 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.203582 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.205772 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:50.205774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:50.205774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:50.205783 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:50.207107 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.207567 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.207568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.207568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.207620 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.207928 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.207929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.207929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.208032 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:50.208106 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.208378 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.208467 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:31:50.208762 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.199995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37227 (local address 127.23.173.194:36787)
0812 14:31:50.200164 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.200217 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.200741 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.200955 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.200960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.201530 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:50.201723 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.201728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.202353 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.202354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.203223 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.203226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.203598 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.203600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.203745 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.204192 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:50.204200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:50.205682 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:50.207187 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:50.207197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:50.207199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:50.207492 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:50.207678 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:50.207679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:50.207680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:50.207865 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:50.208115 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:50.208123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:50.208255 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.208346 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.208504 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260812 14:31:50.209213 23176 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37227
I20260812 14:31:50.209323 23176 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:50.209535 23176 heartbeater.cc:511] Master 127.23.173.254:37227 requested a full tablet report, sending...
I20260812 14:31:50.209918 22619 ts_manager.cc:194] Registered new tserver with Master: 09d3412cef1b4e0684752e61f68de459 (127.23.173.194:38381)
I20260812 14:31:50.210278 22619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:36787
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:31:50.230434 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37227
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:50.312343 23184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:50.312539 23184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:50.312562 23184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:50.313691 23184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:50.313733 23184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:50.313762 23184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:50.315464 23184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:37227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:50.315677 23184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:50.315925 23184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:50.320725 23192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.320778 23184 server_base.cc:1034] running on GCE node
W20260812 14:31:50.320735 23193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:50.320735 23195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.321183 23184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:50.321398 23184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:50.322528 23184 hybrid_clock.cc:648] HybridClock initialized: now 1786545110322516 us; error 29 us; skew 500 ppm
Aug 12 14:31:50 dist-test-slave-1l3l krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545110, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:50.325009 23184 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:31:50.325399 23184 webserver.cc:460] Webserver started at http://127.23.173.195:46397/ using document root <none> and password file <none>
I20260812 14:31:50.325595 23184 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:50.325636 23184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:50.325728 23184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:50.326704 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "57e201083edc43c6b0ecabcff063f9db"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "1788a6888636e462d77e4c97b649d080"
server_key_iv: "30fcbab726f676f77e15715b0d82ec6e"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.326992 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "57e201083edc43c6b0ecabcff063f9db"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "1788a6888636e462d77e4c97b649d080"
server_key_iv: "30fcbab726f676f77e15715b0d82ec6e"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.328202 23184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:50.328826 23202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.328979 23184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:50.329057 23184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "57e201083edc43c6b0ecabcff063f9db"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "1788a6888636e462d77e4c97b649d080"
server_key_iv: "30fcbab726f676f77e15715b0d82ec6e"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.329113 23184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:50.358553 23184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:50.360523 23184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:50.360642 23184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:50.360836 23184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:50.361123 23184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:50.361160 23184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.361189 23184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:50.361217 23184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.366732 23184 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:43157
I20260812 14:31:50.366791 23392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:43157 every 8 connection(s)
I20260812 14:31:50.367074 23184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:31:50 dist-test-slave-1l3l krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545110, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:50.375124 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.367763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:42557 (local address 127.23.173.254:37227)
0812 14:31:50.367826 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.367828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.368000 (+ 172us) server_negotiation.cc:407] Connection header received
0812 14:31:50.368454 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.368455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.368464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.368531 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:50.369405 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.369896 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.370875 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.371062 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.371113 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.373060 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:50.373061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:50.373061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:50.373068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:50.374115 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.374439 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.374440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.374440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.374503 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.374675 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.374675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.374675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.374766 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:50.374811 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.375006 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.375062 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:31:50.375300 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.367738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37227 (local address 127.23.173.195:42557)
0812 14:31:50.367892 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.367928 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.368381 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.368589 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.368592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.369150 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:50.369338 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.369343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.369961 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.369962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.370811 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.370813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.371124 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.371125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.371222 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.371667 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:50.371675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:50.372985 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:50.374180 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:50.374185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:50.374187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:50.374386 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:50.374547 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:50.374548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:50.374548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:50.374634 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:50.374807 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:50.374809 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:50.374913 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.374987 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.375105 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260812 14:31:50.375788 23393 heartbeater.cc:347] Connected to a master server at 127.23.173.254:37227
I20260812 14:31:50.375875 23393 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:50.376065 23393 heartbeater.cc:511] Master 127.23.173.254:37227 requested a full tablet report, sending...
I20260812 14:31:50.376093 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 23184
I20260812 14:31:50.376185 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:31:50.376370 24247 external_mini_cluster.cc:1428] Setting key 3da28ca2ac1cce48fd5466bd9c63faaa
I20260812 14:31:50.376456 22619 ts_manager.cc:194] Registered new tserver with Master: 57e201083edc43c6b0ecabcff063f9db (127.23.173.195:43157)
I20260812 14:31:50.376893 22619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:42557
I20260812 14:31:50.377240 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:50.383944 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.378479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.23.173.254:37227)
0812 14:31:50.378512 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.378514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.378630 (+ 116us) server_negotiation.cc:407] Connection header received
0812 14:31:50.378686 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.378687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.378696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.378740 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:50.379246 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.379738 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.380417 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.380614 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.380665 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.381818 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:50.381819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:50.381820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:50.381826 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:50.382780 (+ 954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.383143 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.383144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.383145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.383193 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:50.383377 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:50.383378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:50.383379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:50.383552 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:50.383648 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.383798 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.383859 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 14:31:50.385645 22618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:50.387166 22618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:50.393141 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.389258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:34969 (local address 127.0.0.1:49976)
0812 14:31:50.389644 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.389652 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.389695 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.389928 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.389930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.389932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:50.389989 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.389991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.390918 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.390919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.392250 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.392252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.392907 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.392908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.392976 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.392986 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.393050 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.393087 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260812 14:31:50.393209 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.389464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.23.173.193:34969)
0812 14:31:50.389654 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.389657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.389692 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:31:50.389747 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.389749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.389763 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.389832 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:50.390068 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.390811 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.392330 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.392825 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.392885 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.392921 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.393085 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.393126 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260812 14:31:50.393833 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.389258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:38381 (local address 127.0.0.1:41350)
0812 14:31:50.389748 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.389754 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.389782 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.390173 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.390173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.390175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:50.390207 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.390208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.391352 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.391353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.392257 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.392259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.393603 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.393604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.393665 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.393670 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.393724 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.393767 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":4,"thread_start_us":42,"threads_started":1}
I20260812 14:31:50.394044 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.389475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.23.173.194:38381)
0812 14:31:50.389889 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.389893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.389902 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:50.389940 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.389942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.389958 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.390027 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:50.390279 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.391269 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.392950 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.393525 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.393592 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.393642 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.393755 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.393803 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":95,"threads_started":1}
I20260812 14:31:50.394039 22837 tablet_service.cc:1467] Processing CreateTablet for tablet afb4919b1d264ffbbcddc5c86fb55bd4 (DEFAULT_TABLE table=test-table [id=00043a4306f44b91a08facb05773a291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:50.394419 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4919b1d264ffbbcddc5c86fb55bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:50.394773 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.389254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:43157 (local address 127.0.0.1:35558)
0812 14:31:50.389401 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.389440 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.389504 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.389812 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.389815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.389817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:50.389921 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.389923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.391171 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.391172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.392248 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.392251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.394601 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.394602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.394664 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.394667 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.394697 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.394725 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":106,"threads_started":1}
I20260812 14:31:50.395035 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.389472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.23.173.195:43157)
0812 14:31:50.389637 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.389642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.389653 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:50.389691 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.389693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.389708 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.389799 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:50.390007 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.391068 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.393678 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.394488 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.394552 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.394912 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.394938 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.394978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260812 14:31:50.395567 23279 tablet_service.cc:1467] Processing CreateTablet for tablet afb4919b1d264ffbbcddc5c86fb55bd4 (DEFAULT_TABLE table=test-table [id=00043a4306f44b91a08facb05773a291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:50.395879 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4919b1d264ffbbcddc5c86fb55bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:50.396183 23070 tablet_service.cc:1467] Processing CreateTablet for tablet afb4919b1d264ffbbcddc5c86fb55bd4 (DEFAULT_TABLE table=test-table [id=00043a4306f44b91a08facb05773a291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:50.396503 23070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4919b1d264ffbbcddc5c86fb55bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:50.397531 23423 tablet_bootstrap.cc:492] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: Bootstrap starting.
I20260812 14:31:50.398083 23423 tablet_bootstrap.cc:654] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:50.398577 23423 log.cc:824] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:50.399116 23429 tablet_bootstrap.cc:492] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Bootstrap starting.
I20260812 14:31:50.399327 23423 tablet_bootstrap.cc:492] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: No bootstrap required, opened a new log
I20260812 14:31:50.399410 23423 ts_tablet_manager.cc:1397] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:50.399778 23429 tablet_bootstrap.cc:654] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:50.400341 23429 log.cc:824] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:50.401324 23429 tablet_bootstrap.cc:492] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: No bootstrap required, opened a new log
I20260812 14:31:50.401404 23429 ts_tablet_manager.cc:1397] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:50.401399 23423 raft_consensus.cc:348] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.401589 23423 raft_consensus.cc:374] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:50.401635 23423 raft_consensus.cc:729] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e3c8fa151f4066b3bc0a79d8e86636, State: Initialized, Role: FOLLOWER
I20260812 14:31:50.401751 23423 consensus_queue.cc:260] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.402040 23423 ts_tablet_manager.cc:1428] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:50.402057 22953 heartbeater.cc:503] Master 127.23.173.254:37227 was elected leader, sending a full tablet report...
I20260812 14:31:50.402261 23430 tablet_bootstrap.cc:492] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Bootstrap starting.
I20260812 14:31:50.402905 23430 tablet_bootstrap.cc:654] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:50.403321 23429 raft_consensus.cc:348] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.403496 23429 raft_consensus.cc:374] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:50.403518 23430 log.cc:824] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:50.403545 23429 raft_consensus.cc:729] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e201083edc43c6b0ecabcff063f9db, State: Initialized, Role: FOLLOWER
I20260812 14:31:50.403657 23429 consensus_queue.cc:260] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.403937 23429 ts_tablet_manager.cc:1428] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:31:50.403986 23393 heartbeater.cc:503] Master 127.23.173.254:37227 was elected leader, sending a full tablet report...
I20260812 14:31:50.404301 23430 tablet_bootstrap.cc:492] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: No bootstrap required, opened a new log
I20260812 14:31:50.404358 23430 ts_tablet_manager.cc:1397] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:31:50.405817 23430 raft_consensus.cc:348] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.405952 23430 raft_consensus.cc:374] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:50.405977 23430 raft_consensus.cc:729] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d3412cef1b4e0684752e61f68de459, State: Initialized, Role: FOLLOWER
I20260812 14:31:50.406070 23430 consensus_queue.cc:260] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.406308 23430 ts_tablet_manager.cc:1428] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:31:50.406402 23176 heartbeater.cc:503] Master 127.23.173.254:37227 was elected leader, sending a full tablet report...
W20260812 14:31:50.449862 23177 tablet.cc:2369] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:31:50.532732 22954 tablet.cc:2369] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:31:50.617615 23394 tablet.cc:2369] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:31:50.658906 23440 raft_consensus.cc:482] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:50.659046 23440 raft_consensus.cc:504] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.659464 23440 leader_election.cc:288] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35e3c8fa151f4066b3bc0a79d8e86636 (127.23.173.193:34969), 57e201083edc43c6b0ecabcff063f9db (127.23.173.195:43157)
I20260812 14:31:50.662426 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.659728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:34969 (local address 127.23.173.194:51495)
0812 14:31:50.659793 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.659805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.659859 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.660049 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.660050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.660052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:50.660126 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.660128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.660775 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.660776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.661629 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.661632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.662157 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.662158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.662252 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.662255 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.662303 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.662347 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 14:31:50.662446 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.659773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:51495 (local address 127.23.173.193:34969)
0812 14:31:50.659842 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.659845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.659858 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:31:50.659942 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.659943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.659954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.660018 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:50.660190 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.660704 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.661722 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.662084 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.662149 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.662159 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.662323 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.662391 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:31:50.662559 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.659730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:43157 (local address 127.23.173.194:60483)
0812 14:31:50.659957 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.659963 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:50.660033 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:50.660167 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:50.660168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:50.660170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:50.660250 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.660251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.660844 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.660845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.661691 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:50.661694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.662299 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.662300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:50.662377 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.662381 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:50.662425 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:31:50.662474 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":28,"thread_start_us":140,"threads_started":1}
I20260812 14:31:50.662545 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.659850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:60483 (local address 127.23.173.195:43157)
0812 14:31:50.659895 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.659896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.660009 (+ 113us) server_negotiation.cc:407] Connection header received
0812 14:31:50.660101 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.660102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.660110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.660157 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:50.660307 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.660772 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.661779 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.662217 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.662276 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.662285 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.662437 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.662477 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:31:50.662784 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4919b1d264ffbbcddc5c86fb55bd4" candidate_uuid: "09d3412cef1b4e0684752e61f68de459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" is_pre_election: true
I20260812 14:31:50.662995 22875 raft_consensus.cc:2454] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d3412cef1b4e0684752e61f68de459 in term 0.
I20260812 14:31:50.663025 23315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4919b1d264ffbbcddc5c86fb55bd4" candidate_uuid: "09d3412cef1b4e0684752e61f68de459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e201083edc43c6b0ecabcff063f9db" is_pre_election: true
I20260812 14:31:50.663185 23315 raft_consensus.cc:2454] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d3412cef1b4e0684752e61f68de459 in term 0.
I20260812 14:31:50.663228 22995 leader_election.cc:302] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d3412cef1b4e0684752e61f68de459, 35e3c8fa151f4066b3bc0a79d8e86636; no voters:
I20260812 14:31:50.663364 23440 raft_consensus.cc:2781] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:50.663436 23440 raft_consensus.cc:482] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:50.663461 23440 raft_consensus.cc:3037] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:50.664268 23440 raft_consensus.cc:504] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.664397 23440 leader_election.cc:288] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [CANDIDATE]: Term 1 election: Requested vote from peers 35e3c8fa151f4066b3bc0a79d8e86636 (127.23.173.193:34969), 57e201083edc43c6b0ecabcff063f9db (127.23.173.195:43157)
I20260812 14:31:50.664561 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4919b1d264ffbbcddc5c86fb55bd4" candidate_uuid: "09d3412cef1b4e0684752e61f68de459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e3c8fa151f4066b3bc0a79d8e86636"
I20260812 14:31:50.664575 23315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4919b1d264ffbbcddc5c86fb55bd4" candidate_uuid: "09d3412cef1b4e0684752e61f68de459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e201083edc43c6b0ecabcff063f9db"
I20260812 14:31:50.664637 22875 raft_consensus.cc:3037] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:50.664654 23315 raft_consensus.cc:3037] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:50.665356 22875 raft_consensus.cc:2454] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d3412cef1b4e0684752e61f68de459 in term 1.
I20260812 14:31:50.665416 23315 raft_consensus.cc:2454] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d3412cef1b4e0684752e61f68de459 in term 1.
I20260812 14:31:50.665581 22995 leader_election.cc:302] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d3412cef1b4e0684752e61f68de459, 35e3c8fa151f4066b3bc0a79d8e86636; no voters:
I20260812 14:31:50.665685 23440 raft_consensus.cc:2781] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:50.665860 23440 raft_consensus.cc:686] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 1 LEADER]: Becoming Leader. State: Replica: 09d3412cef1b4e0684752e61f68de459, State: Running, Role: LEADER
I20260812 14:31:50.665963 23440 consensus_queue.cc:237] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } }
I20260812 14:31:50.666774 22618 catalog_manager.cc:5495] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09d3412cef1b4e0684752e61f68de459 (127.23.173.194). New cstate: current_term: 1 leader_uuid: "09d3412cef1b4e0684752e61f68de459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d3412cef1b4e0684752e61f68de459" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 38381 } health_report { overall_health: HEALTHY } } }
I20260812 14:31:50.674320 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.672263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.23.173.194:38381)
0812 14:31:50.672312 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.672314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.672323 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:50.672380 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.672381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.672398 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.672461 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:50.672552 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.673015 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.673714 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.673910 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.673962 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.674058 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:50.674141 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:50.674206 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.674240 (+ 34us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.674250 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:31:50.675966 23315 raft_consensus.cc:1261] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 1 FOLLOWER]: Refusing update from remote peer 09d3412cef1b4e0684752e61f68de459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:50.675964 22875 raft_consensus.cc:1261] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 1 FOLLOWER]: Refusing update from remote peer 09d3412cef1b4e0684752e61f68de459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:50.676275 23440 consensus_queue.cc:1035] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e3c8fa151f4066b3bc0a79d8e86636" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 34969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:50.676415 23444 consensus_queue.cc:1035] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e201083edc43c6b0ecabcff063f9db" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:50.679129 23450 mvcc.cc:205] Tried to move back new op lower bound from 7317688773326376960 to 7317688773288390656. Current Snapshot: MvccSnapshot[applied={T|T < 7317688773326376960 or (T in {7317688773326376960})}]
I20260812 14:31:50.679277 23451 mvcc.cc:205] Tried to move back new op lower bound from 7317688773326376960 to 7317688773288390656. Current Snapshot: MvccSnapshot[applied={T|T < 7317688773326376960}]
I20260812 14:31:50.681605 22618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:50.681694 22618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:50.682415 22618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 201433eadd974c468516ebe9ce785233: Sending DeleteTablet for 3 replicas of tablet afb4919b1d264ffbbcddc5c86fb55bd4
I20260812 14:31:50.682760 22837 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4919b1d264ffbbcddc5c86fb55bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:50 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:49976
I20260812 14:31:50.682878 23279 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4919b1d264ffbbcddc5c86fb55bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:50 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:35558
I20260812 14:31:50.683071 23465 tablet_replica.cc:330] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636: stopping tablet replica
I20260812 14:31:50.683137 23070 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4919b1d264ffbbcddc5c86fb55bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:50 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:41350
I20260812 14:31:50.683203 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 22728
I20260812 14:31:50.683291 23465 raft_consensus.cc:2229] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:50.683336 23467 tablet_replica.cc:330] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: stopping tablet replica
I20260812 14:31:50.683427 23467 raft_consensus.cc:2229] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:31:50.683537 23467 raft_consensus.cc:2258] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:50.683539 23465 raft_consensus.cc:2258] T afb4919b1d264ffbbcddc5c86fb55bd4 P 35e3c8fa151f4066b3bc0a79d8e86636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:50.684051 23467 ts_tablet_manager.cc:1905] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:50.686101 23467 ts_tablet_manager.cc:1918] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:50.686196 23467 log.cc:1196] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/afb4919b1d264ffbbcddc5c86fb55bd4
I20260812 14:31:50.686483 23467 ts_tablet_manager.cc:1939] T afb4919b1d264ffbbcddc5c86fb55bd4 P 09d3412cef1b4e0684752e61f68de459: Deleting consensus metadata
I20260812 14:31:50.686975 22594 catalog_manager.cc:4841] TS 09d3412cef1b4e0684752e61f68de459 (127.23.173.194:38381): tablet afb4919b1d264ffbbcddc5c86fb55bd4 (table test-table [id=00043a4306f44b91a08facb05773a291]) successfully deleted
I20260812 14:31:50.695935 23466 tablet_replica.cc:330] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: stopping tablet replica
I20260812 14:31:50.696165 23466 raft_consensus.cc:2229] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:50.696349 23466 raft_consensus.cc:2258] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:50.696877 23466 ts_tablet_manager.cc:1905] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:50.698539 23466 ts_tablet_manager.cc:1918] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:50.698619 23466 log.cc:1196] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/afb4919b1d264ffbbcddc5c86fb55bd4
I20260812 14:31:50.698904 23466 ts_tablet_manager.cc:1939] T afb4919b1d264ffbbcddc5c86fb55bd4 P 57e201083edc43c6b0ecabcff063f9db: Deleting consensus metadata
I20260812 14:31:50.699384 22590 catalog_manager.cc:4841] TS 57e201083edc43c6b0ecabcff063f9db (127.23.173.195:43157): tablet afb4919b1d264ffbbcddc5c86fb55bd4 (table test-table [id=00043a4306f44b91a08facb05773a291]) successfully deleted
W20260812 14:31:50.700420 22588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:34969 (error 108)
I20260812 14:31:50.700740 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 22963
I20260812 14:31:50.701287 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.700837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:34969 (local address 127.0.0.1:49988)
0812 14:31:50.701209 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:50.701254 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:34969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351}
W20260812 14:31:50.702560 22588 catalog_manager.cc:4584] TS 35e3c8fa151f4066b3bc0a79d8e86636 (127.23.173.193:34969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afb4919b1d264ffbbcddc5c86fb55bd4: Network error: Client connection negotiation failed: client connection to 127.23.173.193:34969: connect: Connection refused (error 111)
I20260812 14:31:50.714098 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 23184
I20260812 14:31:50.725980 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 22563
2026-08-12T14:31:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2293 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T14:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:50Z Disabled control of system clock
I20260812 14:31:50.760001 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:44065
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:44065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:50.842531 23477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:50.842784 23477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:50.842806 23477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:50.843981 23477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:50.844020 23477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:50.844035 23477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:50.844046 23477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:50.844058 23477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:50.845472 23477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:44065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:50.845698 23477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:50.845961 23477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:50.850946 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:50.850942 23491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.851043 23477 server_base.cc:1034] running on GCE node
W20260812 14:31:50.850942 23488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:50.851471 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:50.851804 23477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:50.852960 23477 hybrid_clock.cc:648] HybridClock initialized: now 1786545110852943 us; error 32 us; skew 500 ppm
I20260812 14:31:50.854215 23477 webserver.cc:460] Webserver started at http://127.23.173.254:32811/ using document root <none> and password file <none>
I20260812 14:31:50.854478 23477 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:50.854533 23477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:50.854664 23477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:50.855826 23477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "ae10897e2ede4d7186b9fff53f5c1075"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "e9b0823579431fd1c622b05942a2807c"
server_key_iv: "ec6405afc7ff00b5571dd443ed333b82"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.856186 23477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "ae10897e2ede4d7186b9fff53f5c1075"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "e9b0823579431fd1c622b05942a2807c"
server_key_iv: "ec6405afc7ff00b5571dd443ed333b82"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.857550 23477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:50.858242 23504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:50.858410 23477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:50.858495 23477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "ae10897e2ede4d7186b9fff53f5c1075"
format_stamp: "Formatted at 2026-08-12 14:31:50 on dist-test-slave-1l3l"
server_key: "e9b0823579431fd1c622b05942a2807c"
server_key_iv: "ec6405afc7ff00b5571dd443ed333b82"
server_key_version: "encryptionkey@0"
I20260812 14:31:50.858557 23477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:50.868713 23477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:50.869035 23477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:50.869168 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:50.873296 23477 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:44065
I20260812 14:31:50.873337 23601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:44065 every 8 connection(s)
I20260812 14:31:50.873678 23477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:50.874679 23602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:50.875978 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 23477
I20260812 14:31:50.876077 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:50.876263 24247 external_mini_cluster.cc:1428] Setting key c39aa81f536935fbec089a736888aa56
I20260812 14:31:50.877355 23602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075: Bootstrap starting.
I20260812 14:31:50.878031 23602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:50.878325 23602 log.cc:824] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:50.879025 23602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075: No bootstrap required, opened a new log
I20260812 14:31:50.881280 23602 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae10897e2ede4d7186b9fff53f5c1075" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44065 } }
I20260812 14:31:50.881445 23602 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:50.881488 23602 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae10897e2ede4d7186b9fff53f5c1075, State: Initialized, Role: FOLLOWER
I20260812 14:31:50.881646 23602 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae10897e2ede4d7186b9fff53f5c1075" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44065 } }
I20260812 14:31:50.881739 23602 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:50.881762 23602 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:50.881785 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:50.876836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.23.173.254:44065)
0812 14:31:50.877028 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:50.877035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:50.877061 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:31:50.877456 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:50.877465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:50.877541 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:50.877662 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:50.877903 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.878811 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.879907 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:50.880325 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:50.880423 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:50.881110 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:50.881113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:50.881113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:50.881281 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:50.881315 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:50.881326 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:50.881376 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:50.881513 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:31:50.881603 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260812 14:31:50.881810 23602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:50.882493 23602 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae10897e2ede4d7186b9fff53f5c1075" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44065 } }
I20260812 14:31:50.882604 23602 leader_election.cc:302] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae10897e2ede4d7186b9fff53f5c1075; no voters:
I20260812 14:31:50.882774 23602 leader_election.cc:288] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:50.882809 23610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:50.883000 23610 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [term 1 LEADER]: Becoming Leader. State: Replica: ae10897e2ede4d7186b9fff53f5c1075, State: Running, Role: LEADER
I20260812 14:31:50.883021 23602 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:50.883100 23610 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae10897e2ede4d7186b9fff53f5c1075" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44065 } }
I20260812 14:31:50.883456 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae10897e2ede4d7186b9fff53f5c1075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae10897e2ede4d7186b9fff53f5c1075" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44065 } } }
I20260812 14:31:50.883553 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:50.883847 23613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae10897e2ede4d7186b9fff53f5c1075. Latest consensus state: current_term: 1 leader_uuid: "ae10897e2ede4d7186b9fff53f5c1075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae10897e2ede4d7186b9fff53f5c1075" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 44065 } } }
I20260812 14:31:50.884045 23613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:50.884176 23626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:50.884639 23626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:50.886248 23626 catalog_manager.cc:1347] Generated new cluster ID: 15d84b2860b84b73bc926c0e85cd3e9d
I20260812 14:31:50.886301 23626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:50.900316 23626 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:50.901062 23626 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:50.910763 23626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075: Generated new TSK 0
I20260812 14:31:50.910983 23626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:31:50.917591 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44065
--builtin_ntp_servers=127.23.173.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:50.999980 23637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:51.000218 23637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:51.000242 23637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:51.001732 23637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:51.001806 23637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:51.001848 23637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:51.003515 23637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:51.003818 23637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:51.004145 23637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:51.010962 23647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:51.010950 23649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:51.010965 23646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:51.011341 23637 server_base.cc:1034] running on GCE node
I20260812 14:31:51.011575 23637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:51.011865 23637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:51.013067 23637 hybrid_clock.cc:648] HybridClock initialized: now 1786545111013039 us; error 39 us; skew 500 ppm
I20260812 14:31:51.014554 23637 webserver.cc:460] Webserver started at http://127.23.173.193:38215/ using document root <none> and password file <none>
I20260812 14:31:51.014789 23637 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:51.014835 23637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:51.014971 23637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:51.016229 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "4b388f65069946848473c4353deaaef5"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "b3030a0f4a6f71fdb4f3510d383aca42"
server_key_iv: "be7d346058185cbe988fdda8e8de405d"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.016582 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "4b388f65069946848473c4353deaaef5"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "b3030a0f4a6f71fdb4f3510d383aca42"
server_key_iv: "be7d346058185cbe988fdda8e8de405d"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.017900 23637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:51.018713 23659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.018925 23637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:51.019019 23637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "4b388f65069946848473c4353deaaef5"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "b3030a0f4a6f71fdb4f3510d383aca42"
server_key_iv: "be7d346058185cbe988fdda8e8de405d"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.019096 23637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:51.032195 23637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:51.032560 23637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:51.032716 23637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:51.032982 23637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:51.033284 23637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:51.033331 23637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.033375 23637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:51.033399 23637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.039311 23637 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:33557
I20260812 14:31:51.039357 23861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:33557 every 8 connection(s)
I20260812 14:31:51.039732 23637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:31:51.043628 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 23637
I20260812 14:31:51.043787 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:31:51.043990 24247 external_mini_cluster.cc:1428] Setting key 9929202560455bd79ed97b271210e068
I20260812 14:31:51.045075 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44065
--builtin_ntp_servers=127.23.173.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:31:51.045534 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.040520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:55461 (local address 127.23.173.254:44065)
0812 14:31:51.040599 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.040601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.040841 (+ 240us) server_negotiation.cc:407] Connection header received
0812 14:31:51.041376 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.041377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.041389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.041436 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:51.042138 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.042651 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.043777 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.044037 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.044115 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.045188 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:51.045190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:51.045190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:51.045196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:51.045217 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:51.045222 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:51.045289 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.045404 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.045434 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:31:51.045728 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.040479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44065 (local address 127.23.173.193:55461)
0812 14:31:51.040702 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.040747 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.041271 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.041514 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.041518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.041526 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:51.042041 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.042046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.042741 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.042742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.043630 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.043633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.044129 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.044130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.044313 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.044974 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:51.044975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:51.045018 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:51.045050 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:51.045319 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:51.045326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:51.045329 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.045390 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.045509 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260812 14:31:51.046245 23862 heartbeater.cc:347] Connected to a master server at 127.23.173.254:44065
I20260812 14:31:51.046339 23862 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:51.046522 23862 heartbeater.cc:511] Master 127.23.173.254:44065 requested a full tablet report, sending...
I20260812 14:31:51.047061 23532 ts_manager.cc:194] Registered new tserver with Master: 4b388f65069946848473c4353deaaef5 (127.23.173.193:33557)
I20260812 14:31:51.047744 23532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.193:55461
W20260812 14:31:51.129851 23866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:51.130079 23866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:51.130102 23866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:51.131306 23866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:51.131351 23866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:51.131377 23866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:51.132803 23866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:51.133055 23866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:51.133369 23866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:51.138479 23879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:51.138592 23866 server_base.cc:1034] running on GCE node
W20260812 14:31:51.138491 23884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:51.138491 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:51.138901 23866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:51.139143 23866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:51.140280 23866 hybrid_clock.cc:648] HybridClock initialized: now 1786545111140267 us; error 30 us; skew 500 ppm
I20260812 14:31:51.141525 23866 webserver.cc:460] Webserver started at http://127.23.173.194:33795/ using document root <none> and password file <none>
I20260812 14:31:51.141760 23866 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:51.141805 23866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:51.141934 23866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:51.143016 23866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "5e502e59f8744de4945d9dc813c85adf"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "6dc4304445ff739a173419aea2e89a85"
server_key_iv: "3b590229d81de91f5af1971f264f7585"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.143350 23866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "5e502e59f8744de4945d9dc813c85adf"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "6dc4304445ff739a173419aea2e89a85"
server_key_iv: "3b590229d81de91f5af1971f264f7585"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.144667 23866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:51.145342 23894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.145488 23866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:51.145571 23866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "5e502e59f8744de4945d9dc813c85adf"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "6dc4304445ff739a173419aea2e89a85"
server_key_iv: "3b590229d81de91f5af1971f264f7585"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.145635 23866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:51.156155 23866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:51.156473 23866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:51.156617 23866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:51.156840 23866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:51.157157 23866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:51.157193 23866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.157224 23866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:51.157249 23866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.163041 23866 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:44061
I20260812 14:31:51.163100 24126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:44061 every 8 connection(s)
I20260812 14:31:51.163455 23866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:31:51.168720 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.164181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:55643 (local address 127.23.173.254:44065)
0812 14:31:51.164243 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.164245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.164488 (+ 243us) server_negotiation.cc:407] Connection header received
0812 14:31:51.165027 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.165029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.165042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.165100 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:51.165801 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.166309 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.167283 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.167492 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.167552 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.168435 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:51.168436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:51.168437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:51.168442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:51.168459 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:51.168463 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:51.168509 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.168607 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.168636 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:31:51.168927 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.164183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44065 (local address 127.23.173.194:55643)
0812 14:31:51.164369 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.164403 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.164922 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.165180 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.165184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.165191 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:51.165711 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.165716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.166399 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.166400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.167198 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.167200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.167577 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.167578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.167733 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.168286 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:51.168286 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:51.168319 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:51.168345 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:51.168515 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:51.168521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:51.168524 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.168598 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.168709 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260812 14:31:51.169456 24127 heartbeater.cc:347] Connected to a master server at 127.23.173.254:44065
I20260812 14:31:51.169539 24127 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:51.169745 24127 heartbeater.cc:511] Master 127.23.173.254:44065 requested a full tablet report, sending...
I20260812 14:31:51.170094 23532 ts_manager.cc:194] Registered new tserver with Master: 5e502e59f8744de4945d9dc813c85adf (127.23.173.194:44061)
I20260812 14:31:51.170449 23532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.194:55643
I20260812 14:31:51.171283 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 23866
I20260812 14:31:51.171378 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:31:51.171568 24247 external_mini_cluster.cc:1428] Setting key 47ee1a6e6fd559b03d1e338488c2b0af
I20260812 14:31:51.172451 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44065
--builtin_ntp_servers=127.23.173.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:51.254745 24132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:51.254977 24132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:51.255009 24132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:51.256261 24132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:51.256304 24132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:51.256330 24132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:51.257696 24132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:51.257929 24132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:51.258229 24132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:51.263743 24143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:51.263743 24144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:51.263741 24148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:51.263808 24132 server_base.cc:1034] running on GCE node
I20260812 14:31:51.264164 24132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:51.264424 24132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:51.265573 24132 hybrid_clock.cc:648] HybridClock initialized: now 1786545111265556 us; error 35 us; skew 500 ppm
I20260812 14:31:51.266733 24132 webserver.cc:460] Webserver started at http://127.23.173.195:44867/ using document root <none> and password file <none>
I20260812 14:31:51.266947 24132 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:51.266992 24132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:51.267115 24132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:51.268266 24132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "f6fbd2684348472e9af533a00c304aa1"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "adfbc8709319005c6304c9af352fbc4a"
server_key_iv: "22a781f7ffe9e619fe4605fb7d6dc38f"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.268630 24132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "f6fbd2684348472e9af533a00c304aa1"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "adfbc8709319005c6304c9af352fbc4a"
server_key_iv: "22a781f7ffe9e619fe4605fb7d6dc38f"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.269937 24132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:51.270689 24163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.270856 24132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:51.270941 24132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "f6fbd2684348472e9af533a00c304aa1"
format_stamp: "Formatted at 2026-08-12 14:31:51 on dist-test-slave-1l3l"
server_key: "adfbc8709319005c6304c9af352fbc4a"
server_key_iv: "22a781f7ffe9e619fe4605fb7d6dc38f"
server_key_version: "encryptionkey@0"
I20260812 14:31:51.271003 24132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:51.281780 24132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:51.282058 24132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:51.282188 24132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:51.282433 24132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:51.282771 24132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:51.282806 24132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.282835 24132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:51.282860 24132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:51.288654 24132 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:39679
I20260812 14:31:51.288719 24384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:39679 every 8 connection(s)
I20260812 14:31:51.289021 24132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:31:51.293867 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.289695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:41755 (local address 127.23.173.254:44065)
0812 14:31:51.289754 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.289756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.289988 (+ 232us) server_negotiation.cc:407] Connection header received
0812 14:31:51.290552 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.290554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.290568 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.290609 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:51.291265 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.291797 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.292737 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.292914 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.292955 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.293667 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:51.293669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:51.293669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:51.293673 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:51.293687 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:51.293690 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:51.293727 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.293784 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.293807 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:31:51.294066 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.289667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:44065 (local address 127.23.173.195:41755)
0812 14:31:51.289862 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.289906 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.290446 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.290677 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.290681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.290689 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:51.291200 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.291205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.291863 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.291863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.292672 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.292674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.292971 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.292971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.293090 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.293543 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:51.293543 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:51.293571 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:51.293594 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:51.293729 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:51.293731 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:51.293734 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.293777 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.293856 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260812 14:31:51.294543 24387 heartbeater.cc:347] Connected to a master server at 127.23.173.254:44065
I20260812 14:31:51.294633 24387 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:51.294785 24387 heartbeater.cc:511] Master 127.23.173.254:44065 requested a full tablet report, sending...
I20260812 14:31:51.295120 23532 ts_manager.cc:194] Registered new tserver with Master: f6fbd2684348472e9af533a00c304aa1 (127.23.173.195:39679)
I20260812 14:31:51.295480 23532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.195:41755
I20260812 14:31:51.298002 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24132
I20260812 14:31:51.298094 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:31:51.298254 24247 external_mini_cluster.cc:1428] Setting key 87d1e25ab9332a76492ee3851f059660
I20260812 14:31:51.299002 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:51.303032 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.300257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.23.173.254:44065)
0812 14:31:51.300299 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.300300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.300419 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:31:51.300468 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.300469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.300479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.300519 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:51.300693 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.301154 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.301873 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.302053 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.302108 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.302761 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:51.302763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:51.302763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:51.302768 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:51.302782 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:51.302786 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:51.302863 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.302939 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.302950 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:31:51.304531 23532 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:51.305999 23532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:51.312126 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.308394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.23.173.194:44061)
0812 14:31:51.308563 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.308568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.308579 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:51.308621 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.308623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.308639 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.308754 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:51.308926 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.309994 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.311189 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.311689 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.311815 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.311916 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.311995 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.312035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260812 14:31:51.312114 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.307990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:44061 (local address 127.0.0.1:42278)
0812 14:31:51.308136 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.308179 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.308247 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.308765 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.308768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.308771 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:51.308836 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.308839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.310075 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.310076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.311105 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.311106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.311815 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.311816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.311895 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.311908 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.311980 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.312041 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260812 14:31:51.312381 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.308188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:39679 (local address 127.0.0.1:39510)
0812 14:31:51.308388 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.308397 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.308442 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.309105 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.309106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.309108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:51.309146 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.309148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.310317 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.310318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.311174 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.311177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.312157 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.312158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.312217 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.312221 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.312271 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.312309 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":16,"thread_start_us":63,"threads_started":1}
I20260812 14:31:51.312525 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.308003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33557 (local address 127.0.0.1:35654)
0812 14:31:51.308165 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.308192 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.308245 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.308841 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.308842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.308844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:51.308890 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.308892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.309839 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.309840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.311073 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.311075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.312069 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.312070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.312244 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.312247 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.312427 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.312467 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260812 14:31:51.312662 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.308198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.23.173.193:33557)
0812 14:31:51.308635 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.308639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.308648 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:51.308684 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.308686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.308700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.308767 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:51.308967 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.309765 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.311171 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.311928 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.312139 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.312353 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.312458 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.312537 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":89,"threads_started":1}
I20260812 14:31:51.312680 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.308365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.23.173.195:39679)
0812 14:31:51.308892 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.308896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.308904 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:31:51.308940 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.308943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.308958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.309037 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:51.309232 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.310242 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.311262 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.311980 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.312040 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.312125 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.312544 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.312602 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":111,"threads_started":1}
I20260812 14:31:51.313201 23998 tablet_service.cc:1467] Processing CreateTablet for tablet 8b3ba138ffdf436bb89ca2c3a9af890c (DEFAULT_TABLE table=test-table [id=0306a6af20294fbcbdcc6041c9f852d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:51.313288 24259 tablet_service.cc:1467] Processing CreateTablet for tablet 8b3ba138ffdf436bb89ca2c3a9af890c (DEFAULT_TABLE table=test-table [id=0306a6af20294fbcbdcc6041c9f852d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:51.313555 23998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b3ba138ffdf436bb89ca2c3a9af890c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:51.313661 24259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b3ba138ffdf436bb89ca2c3a9af890c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:51.313999 23748 tablet_service.cc:1467] Processing CreateTablet for tablet 8b3ba138ffdf436bb89ca2c3a9af890c (DEFAULT_TABLE table=test-table [id=0306a6af20294fbcbdcc6041c9f852d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:51.314419 23748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b3ba138ffdf436bb89ca2c3a9af890c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:51.316442 24406 tablet_bootstrap.cc:492] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Bootstrap starting.
I20260812 14:31:51.317003 24407 tablet_bootstrap.cc:492] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Bootstrap starting.
I20260812 14:31:51.317083 24406 tablet_bootstrap.cc:654] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:51.317413 24406 log.cc:824] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:51.317736 24407 tablet_bootstrap.cc:654] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:51.318050 24407 log.cc:824] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:51.318284 24406 tablet_bootstrap.cc:492] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: No bootstrap required, opened a new log
I20260812 14:31:51.318352 24406 ts_tablet_manager.cc:1397] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:51.318751 24407 tablet_bootstrap.cc:492] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: No bootstrap required, opened a new log
I20260812 14:31:51.318825 24407 ts_tablet_manager.cc:1397] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:51.319859 24409 tablet_bootstrap.cc:492] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5: Bootstrap starting.
I20260812 14:31:51.320377 24409 tablet_bootstrap.cc:654] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:51.320845 24409 log.cc:824] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:51.320923 24406 raft_consensus.cc:348] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.321103 24406 raft_consensus.cc:374] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:51.321148 24406 raft_consensus.cc:729] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e502e59f8744de4945d9dc813c85adf, State: Initialized, Role: FOLLOWER
I20260812 14:31:51.321265 24406 consensus_queue.cc:260] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.321281 24407 raft_consensus.cc:348] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.321421 24407 raft_consensus.cc:374] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:51.321467 24407 raft_consensus.cc:729] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fbd2684348472e9af533a00c304aa1, State: Initialized, Role: FOLLOWER
I20260812 14:31:51.321544 24406 ts_tablet_manager.cc:1428] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:51.321585 24409 tablet_bootstrap.cc:492] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5: No bootstrap required, opened a new log
I20260812 14:31:51.321578 24407 consensus_queue.cc:260] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.321645 24409 ts_tablet_manager.cc:1397] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:51.321662 24127 heartbeater.cc:503] Master 127.23.173.254:44065 was elected leader, sending a full tablet report...
I20260812 14:31:51.321815 24407 ts_tablet_manager.cc:1428] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:51.321936 24387 heartbeater.cc:503] Master 127.23.173.254:44065 was elected leader, sending a full tablet report...
I20260812 14:31:51.323683 24409 raft_consensus.cc:348] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.323860 24409 raft_consensus.cc:374] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:51.323895 24409 raft_consensus.cc:729] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b388f65069946848473c4353deaaef5, State: Initialized, Role: FOLLOWER
I20260812 14:31:51.323998 24409 consensus_queue.cc:260] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.324229 24409 ts_tablet_manager.cc:1428] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:31:51.324286 23862 heartbeater.cc:503] Master 127.23.173.254:44065 was elected leader, sending a full tablet report...
I20260812 14:31:51.343854 24414 raft_consensus.cc:482] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:51.343948 24414 raft_consensus.cc:504] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.344257 24414 leader_election.cc:288] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e502e59f8744de4945d9dc813c85adf (127.23.173.194:44061), 4b388f65069946848473c4353deaaef5 (127.23.173.193:33557)
I20260812 14:31:51.347743 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.344445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:49897 (local address 127.23.173.194:44061)
0812 14:31:51.344483 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.344485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.344492 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:31:51.344556 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.344557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.344566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.344607 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:51.344765 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.345505 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.346841 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.347374 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.347443 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.347451 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.347617 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.347655 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:31:51.347749 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.344540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:52179 (local address 127.23.173.193:33557)
0812 14:31:51.344581 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.344583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.344689 (+ 106us) server_negotiation.cc:407] Connection header received
0812 14:31:51.344731 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.344732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.344742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.344794 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:51.344915 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.345604 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.346838 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.347374 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.347443 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.347451 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.347617 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.347653 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:31:51.347769 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.344392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:44061 (local address 127.23.173.195:49897)
0812 14:31:51.344442 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.344452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.344490 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.344630 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.344631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.344632 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:51.344702 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.344704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.345575 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.345576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.346751 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.346754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.347446 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.347447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.347547 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.347554 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.347607 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.347651 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":37}
I20260812 14:31:51.347779 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.344412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33557 (local address 127.23.173.195:52179)
0812 14:31:51.344617 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.344625 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:51.344685 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:51.344807 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:51.344807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:51.344809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:51.344863 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.344864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.345665 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.345666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.346754 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:51.346756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.347449 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.347449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:51.347547 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.347554 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:31:51.347607 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:31:51.347651 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":7,"thread_start_us":111,"threads_started":1}
I20260812 14:31:51.348089 24032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3ba138ffdf436bb89ca2c3a9af890c" candidate_uuid: "f6fbd2684348472e9af533a00c304aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e502e59f8744de4945d9dc813c85adf" is_pre_election: true
I20260812 14:31:51.348165 23782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3ba138ffdf436bb89ca2c3a9af890c" candidate_uuid: "f6fbd2684348472e9af533a00c304aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b388f65069946848473c4353deaaef5" is_pre_election: true
I20260812 14:31:51.348243 24032 raft_consensus.cc:2454] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6fbd2684348472e9af533a00c304aa1 in term 0.
I20260812 14:31:51.348285 23782 raft_consensus.cc:2454] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6fbd2684348472e9af533a00c304aa1 in term 0.
I20260812 14:31:51.348496 24164 leader_election.cc:302] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b388f65069946848473c4353deaaef5, f6fbd2684348472e9af533a00c304aa1; no voters:
I20260812 14:31:51.348640 24414 raft_consensus.cc:2781] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:51.348706 24414 raft_consensus.cc:482] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:51.348722 24414 raft_consensus.cc:3037] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:51.349464 24414 raft_consensus.cc:504] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.349599 24414 leader_election.cc:288] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [CANDIDATE]: Term 1 election: Requested vote from peers 5e502e59f8744de4945d9dc813c85adf (127.23.173.194:44061), 4b388f65069946848473c4353deaaef5 (127.23.173.193:33557)
I20260812 14:31:51.349769 23782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3ba138ffdf436bb89ca2c3a9af890c" candidate_uuid: "f6fbd2684348472e9af533a00c304aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b388f65069946848473c4353deaaef5"
I20260812 14:31:51.349767 24032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3ba138ffdf436bb89ca2c3a9af890c" candidate_uuid: "f6fbd2684348472e9af533a00c304aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e502e59f8744de4945d9dc813c85adf"
I20260812 14:31:51.349844 24032 raft_consensus.cc:3037] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:51.349844 23782 raft_consensus.cc:3037] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:51.350699 24032 raft_consensus.cc:2454] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6fbd2684348472e9af533a00c304aa1 in term 1.
I20260812 14:31:51.350699 23782 raft_consensus.cc:2454] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6fbd2684348472e9af533a00c304aa1 in term 1.
I20260812 14:31:51.350852 24164 leader_election.cc:302] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b388f65069946848473c4353deaaef5, f6fbd2684348472e9af533a00c304aa1; no voters:
I20260812 14:31:51.350927 24414 raft_consensus.cc:2781] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:51.351063 24414 raft_consensus.cc:686] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 1 LEADER]: Becoming Leader. State: Replica: f6fbd2684348472e9af533a00c304aa1, State: Running, Role: LEADER
I20260812 14:31:51.351141 24414 consensus_queue.cc:237] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } }
I20260812 14:31:51.351877 23531 catalog_manager.cc:5495] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6fbd2684348472e9af533a00c304aa1 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "f6fbd2684348472e9af533a00c304aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6fbd2684348472e9af533a00c304aa1" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 39679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 } health_report { overall_health: UNKNOWN } } }
I20260812 14:31:51.357823 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.355857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.23.173.195:39679)
0812 14:31:51.355899 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:51.355901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:51.355927 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:31:51.355975 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:51.355975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:51.355995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:51.356043 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:51.356140 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.356584 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.357270 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:51.357433 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:51.357472 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:51.357594 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:51.357671 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:51.357735 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:51.357757 (+ 22us) server_negotiation.cc:299] Negotiation successful
0812 14:31:51.357764 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:31:51.359326 23782 raft_consensus.cc:1261] T 8b3ba138ffdf436bb89ca2c3a9af890c P 4b388f65069946848473c4353deaaef5 [term 1 FOLLOWER]: Refusing update from remote peer f6fbd2684348472e9af533a00c304aa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:51.359337 24032 raft_consensus.cc:1261] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 1 FOLLOWER]: Refusing update from remote peer f6fbd2684348472e9af533a00c304aa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:51.359650 24414 consensus_queue.cc:1035] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e502e59f8744de4945d9dc813c85adf" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:51.359797 24418 consensus_queue.cc:1035] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b388f65069946848473c4353deaaef5" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:51.362941 24425 mvcc.cc:205] Tried to move back new op lower bound from 7317688776125825024 to 7317688776094900224. Current Snapshot: MvccSnapshot[applied={T|T < 7317688776125825024}]
I20260812 14:31:51.363054 24424 mvcc.cc:205] Tried to move back new op lower bound from 7317688776125825024 to 7317688776094900224. Current Snapshot: MvccSnapshot[applied={T|T < 7317688776125825024}]
I20260812 14:31:51.364499 23532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:51.364578 23532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:51.365226 23532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae10897e2ede4d7186b9fff53f5c1075: Sending DeleteTablet for 3 replicas of tablet 8b3ba138ffdf436bb89ca2c3a9af890c
I20260812 14:31:51.365562 23748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b3ba138ffdf436bb89ca2c3a9af890c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:51 UTC) from {username='slave'} at 127.0.0.1:35654
I20260812 14:31:51.365597 23998 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b3ba138ffdf436bb89ca2c3a9af890c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:51 UTC) from {username='slave'} at 127.0.0.1:42278
I20260812 14:31:51.365816 24437 tablet_replica.cc:330] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: stopping tablet replica
I20260812 14:31:51.365880 24259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b3ba138ffdf436bb89ca2c3a9af890c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:51 UTC) from {username='slave'} at 127.0.0.1:39510
I20260812 14:31:51.365917 24437 raft_consensus.cc:2229] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:51.365986 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 23637
I20260812 14:31:51.366004 24437 raft_consensus.cc:2258] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:51.366142 24438 tablet_replica.cc:330] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: stopping tablet replica
I20260812 14:31:51.366271 24438 raft_consensus.cc:2229] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:31:51.366385 24438 raft_consensus.cc:2258] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:51.404700 24438 ts_tablet_manager.cc:1905] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:51.404815 24437 ts_tablet_manager.cc:1905] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:51.406725 24438 ts_tablet_manager.cc:1918] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:51.406733 24437 ts_tablet_manager.cc:1918] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:51.406806 24437 log.cc:1196] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/8b3ba138ffdf436bb89ca2c3a9af890c
I20260812 14:31:51.406963 24438 log.cc:1196] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/8b3ba138ffdf436bb89ca2c3a9af890c
I20260812 14:31:51.407084 24437 ts_tablet_manager.cc:1939] T 8b3ba138ffdf436bb89ca2c3a9af890c P 5e502e59f8744de4945d9dc813c85adf: Deleting consensus metadata
I20260812 14:31:51.407368 24438 ts_tablet_manager.cc:1939] T 8b3ba138ffdf436bb89ca2c3a9af890c P f6fbd2684348472e9af533a00c304aa1: Deleting consensus metadata
I20260812 14:31:51.407876 23510 catalog_manager.cc:4841] TS 5e502e59f8744de4945d9dc813c85adf (127.23.173.194:44061): tablet 8b3ba138ffdf436bb89ca2c3a9af890c (table test-table [id=0306a6af20294fbcbdcc6041c9f852d2]) successfully deleted
I20260812 14:31:51.408001 23510 catalog_manager.cc:4841] TS f6fbd2684348472e9af533a00c304aa1 (127.23.173.195:39679): tablet 8b3ba138ffdf436bb89ca2c3a9af890c (table test-table [id=0306a6af20294fbcbdcc6041c9f852d2]) successfully deleted
W20260812 14:31:51.410097 23507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:33557 (error 108)
I20260812 14:31:51.410322 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 23866
I20260812 14:31:51.412595 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:51.410551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:33557 (local address 127.0.0.1:35668)
0812 14:31:51.412502 (+ 1951us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:51.412557 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1926}
W20260812 14:31:51.416108 23507 catalog_manager.cc:4584] TS 4b388f65069946848473c4353deaaef5 (127.23.173.193:33557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b3ba138ffdf436bb89ca2c3a9af890c: Network error: Client connection negotiation failed: client connection to 127.23.173.193:33557: connect: Connection refused (error 111)
I20260812 14:31:51.420827 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24132
I20260812 14:31:51.439767 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 23477
2026-08-12T14:31:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (714 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:51 dist-test-slave-1l3l krb5kdc[24446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:51 dist-test-slave-1l3l krb5kdc[24446](info): set up 2 sockets
Aug 12 14:31:51 dist-test-slave-1l3l krb5kdc[24446](info): commencing operation
krb5kdc: starting...
W20260812 14:31:52.618294 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:52 dist-test-slave-1l3l krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:31:52.739732 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43215
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:52.823964 24467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:52.824190 24467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:52.824214 24467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:52.825459 24467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:52.825506 24467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:52.825519 24467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:52.825531 24467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:52.825541 24467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:52.827069 24467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:43215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:52.827323 24467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:52.827585 24467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:52.832828 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:52.832828 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:52.833004 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:52.833041 24467 server_base.cc:1034] running on GCE node
I20260812 14:31:52.833441 24467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:52.833753 24467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:52.834936 24467 hybrid_clock.cc:648] HybridClock initialized: now 1786545112834914 us; error 36 us; skew 500 ppm
Aug 12 14:31:52 dist-test-slave-1l3l krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545112, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:52.837689 24467 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:31:52.838186 24467 webserver.cc:460] Webserver started at http://127.23.173.254:45151/ using document root <none> and password file <none>
I20260812 14:31:52.838413 24467 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:52.838466 24467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:52.838601 24467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:52.839758 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "c59fd4feafec464596eeed16fd72a4d6"
format_stamp: "Formatted at 2026-08-12 14:31:52 on dist-test-slave-1l3l"
server_key: "671754287314c7e3399749dc0eaf88e2"
server_key_iv: "d37d146542a065b6b3a1e25402ad50d2"
server_key_version: "encryptionkey@0"
I20260812 14:31:52.840140 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "c59fd4feafec464596eeed16fd72a4d6"
format_stamp: "Formatted at 2026-08-12 14:31:52 on dist-test-slave-1l3l"
server_key: "671754287314c7e3399749dc0eaf88e2"
server_key_iv: "d37d146542a065b6b3a1e25402ad50d2"
server_key_version: "encryptionkey@0"
I20260812 14:31:52.841531 24467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:52.842294 24495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:52.842445 24467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:52.842532 24467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "c59fd4feafec464596eeed16fd72a4d6"
format_stamp: "Formatted at 2026-08-12 14:31:52 on dist-test-slave-1l3l"
server_key: "671754287314c7e3399749dc0eaf88e2"
server_key_iv: "d37d146542a065b6b3a1e25402ad50d2"
server_key_version: "encryptionkey@0"
I20260812 14:31:52.842604 24467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:52.862666 24467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:52.864977 24467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:52.865128 24467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:52.869230 24467 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:43215
I20260812 14:31:52.869268 24591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:43215 every 8 connection(s)
I20260812 14:31:52.869611 24467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:52.870450 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:52.873206 24594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6: Bootstrap starting.
I20260812 14:31:52.873951 24594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:52.874295 24594 log.cc:824] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:52.875195 24594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6: No bootstrap required, opened a new log
I20260812 14:31:52.875633 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24467
I20260812 14:31:52.875751 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:52.875934 24247 external_mini_cluster.cc:1428] Setting key 4d3d7e02593eedc913bd63f62485a2c8
I20260812 14:31:52.877722 24594 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fd4feafec464596eeed16fd72a4d6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43215 } }
I20260812 14:31:52.877871 24594 raft_consensus.cc:374] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:52.877903 24594 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59fd4feafec464596eeed16fd72a4d6, State: Initialized, Role: FOLLOWER
I20260812 14:31:52.878026 24594 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fd4feafec464596eeed16fd72a4d6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43215 } }
I20260812 14:31:52.878103 24594 raft_consensus.cc:388] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:52.878135 24594 raft_consensus.cc:482] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:52.878182 24594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:52.878768 24594 raft_consensus.cc:504] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fd4feafec464596eeed16fd72a4d6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43215 } }
I20260812 14:31:52.878863 24594 leader_election.cc:302] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59fd4feafec464596eeed16fd72a4d6; no voters:
I20260812 14:31:52.879019 24594 leader_election.cc:288] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:52.879105 24607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:52.879287 24607 raft_consensus.cc:686] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [term 1 LEADER]: Becoming Leader. State: Replica: c59fd4feafec464596eeed16fd72a4d6, State: Running, Role: LEADER
I20260812 14:31:52.879303 24594 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:52.879482 24607 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fd4feafec464596eeed16fd72a4d6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43215 } }
I20260812 14:31:52.879943 24609 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59fd4feafec464596eeed16fd72a4d6. Latest consensus state: current_term: 1 leader_uuid: "c59fd4feafec464596eeed16fd72a4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fd4feafec464596eeed16fd72a4d6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43215 } } }
I20260812 14:31:52.879956 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c59fd4feafec464596eeed16fd72a4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fd4feafec464596eeed16fd72a4d6" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43215 } } }
I20260812 14:31:52.880043 24609 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:52.880059 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:52.880296 24619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:52.880822 24619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:31:52 dist-test-slave-1l3l krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:52.887619 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:52.876455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.23.173.254:43215)
0812 14:31:52.876660 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:52.876668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:52.876694 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:31:52.877650 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:52.877658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:52.877713 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:52.877846 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:52.878552 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:52.879675 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:52.880717 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:52.881327 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:52.881429 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:52.884259 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:52.884264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:52.884265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:52.884279 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:52.886098 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:52.886422 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:52.886426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:52.886427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:52.886539 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:52.886695 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:52.886696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:52.886696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:52.887028 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:52.887181 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:52.887253 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:31:52.887408 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260812 14:31:52.887907 24619 catalog_manager.cc:1347] Generated new cluster ID: 2fc4bd1bbf154967ad90ec47e4ef8396
I20260812 14:31:52.887956 24619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:52.903456 24619 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:52.904309 24619 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:52.913406 24619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6: Generated new TSK 0
I20260812 14:31:52.913645 24619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
Entry for principal kudu/127.23.173.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab.
I20260812 14:31:52.950181 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:43215
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:53.034407 24637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:53.034602 24637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:53.034624 24637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:53.035794 24637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:53.035833 24637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:53.035858 24637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:53.037277 24637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.193.keytab
--principal=kudu/127.23.173.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:43215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:53.037510 24637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:53.037724 24637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:53.042659 24649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:53.042665 24644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:53.042670 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.042876 24637 server_base.cc:1034] running on GCE node
I20260812 14:31:53.043058 24637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:53.043275 24637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:53.044458 24637 hybrid_clock.cc:648] HybridClock initialized: now 1786545113044441 us; error 40 us; skew 500 ppm
Aug 12 14:31:53 dist-test-slave-1l3l krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:53.047031 24637 init.cc:377] Logged in from keytab as kudu/127.23.173.193@KRBTEST.COM (short username kudu)
I20260812 14:31:53.047493 24637 webserver.cc:460] Webserver started at http://127.23.173.193:43837/ using document root <none> and password file <none>
I20260812 14:31:53.047767 24637 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:53.047827 24637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:53.047956 24637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:53.048974 24637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "34e4bddf6f424e1e8019482e67a15ea2"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "7984d77ff59a626641f6e80ff77359cd"
server_key_iv: "61482d8da9725c398d0905db8026b46e"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.049376 24637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "34e4bddf6f424e1e8019482e67a15ea2"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "7984d77ff59a626641f6e80ff77359cd"
server_key_iv: "61482d8da9725c398d0905db8026b46e"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.050594 24637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:53.051244 24660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.051393 24637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:53.051482 24637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "34e4bddf6f424e1e8019482e67a15ea2"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "7984d77ff59a626641f6e80ff77359cd"
server_key_iv: "61482d8da9725c398d0905db8026b46e"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.051541 24637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:53.063953 24637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:53.065804 24637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:53.065950 24637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:53.066177 24637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:53.066501 24637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:53.066537 24637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.066567 24637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:53.066584 24637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.072291 24637 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:42953
I20260812 14:31:53.072352 24863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:42953 every 8 connection(s)
I20260812 14:31:53.072675 24637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:31:53.076424 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24637
I20260812 14:31:53.076514 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:31:53.076700 24247 external_mini_cluster.cc:1428] Setting key 53aefd55dfb0484c6bdcc225dd5973e7
Aug 12 14:31:53 dist-test-slave-1l3l krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.193@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:53.082321 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.073355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:34841 (local address 127.23.173.254:43215)
0812 14:31:53.073442 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.073445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.073608 (+ 163us) server_negotiation.cc:407] Connection header received
0812 14:31:53.074109 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.074110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.074120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.074171 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:53.075118 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.075653 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.076772 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.077018 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.077076 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.079499 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:53.079500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:53.079500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:53.079508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:53.081001 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.081381 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.081382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.081383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.081450 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.081657 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.081658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.081658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.081792 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:53.081872 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.082119 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.082216 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 14:31:53.082427 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.073316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:43215 (local address 127.23.173.193:34841)
0812 14:31:53.073486 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.073527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.074025 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.074228 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.074231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.074807 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:53.075027 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.075033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.075754 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.075756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.076682 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.076686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.077096 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.077097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.077251 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.077844 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:53.077853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:53.079399 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:53.081088 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:53.081095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:53.081097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:53.081313 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:53.081508 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:53.081509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:53.081509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:53.081606 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:53.081855 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:53.081859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:53.081987 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.082066 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.082232 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260812 14:31:53.082935 24868 heartbeater.cc:347] Connected to a master server at 127.23.173.254:43215
I20260812 14:31:53.083038 24868 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:53.083220 24868 heartbeater.cc:511] Master 127.23.173.254:43215 requested a full tablet report, sending...
I20260812 14:31:53.083850 24526 ts_manager.cc:194] Registered new tserver with Master: 34e4bddf6f424e1e8019482e67a15ea2 (127.23.173.193:42953)
I20260812 14:31:53.084745 24526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.193@KRBTEST.COM'} at 127.23.173.193:34841
WARNING: no policy specified for kudu/127.23.173.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
Entry for principal kudu/127.23.173.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab.
I20260812 14:31:53.102880 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:43215
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:53.184455 24875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:53.184664 24875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:53.184686 24875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:53.185828 24875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:53.185868 24875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:53.185895 24875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:53.187321 24875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.194.keytab
--principal=kudu/127.23.173.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:43215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:53.187553 24875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:53.187774 24875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:53.192672 24881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:53.192673 24884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.192756 24875 server_base.cc:1034] running on GCE node
W20260812 14:31:53.192687 24882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.193049 24875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:53.193295 24875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:53.194479 24875 hybrid_clock.cc:648] HybridClock initialized: now 1786545113194457 us; error 32 us; skew 500 ppm
Aug 12 14:31:53 dist-test-slave-1l3l krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:53.196887 24875 init.cc:377] Logged in from keytab as kudu/127.23.173.194@KRBTEST.COM (short username kudu)
I20260812 14:31:53.197295 24875 webserver.cc:460] Webserver started at http://127.23.173.194:46695/ using document root <none> and password file <none>
I20260812 14:31:53.197513 24875 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:53.197563 24875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:53.197705 24875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:53.198697 24875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "a7ab2a0e74874cc9a0437cf6730c52fd"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "f80915d3862c798c07acbd10b611d37d"
server_key_iv: "82ae0430c7e1ba830557bb253caa2225"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.199023 24875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "a7ab2a0e74874cc9a0437cf6730c52fd"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "f80915d3862c798c07acbd10b611d37d"
server_key_iv: "82ae0430c7e1ba830557bb253caa2225"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.200335 24875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:53.201395 24895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.201561 24875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:31:53.201642 24875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "a7ab2a0e74874cc9a0437cf6730c52fd"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "f80915d3862c798c07acbd10b611d37d"
server_key_iv: "82ae0430c7e1ba830557bb253caa2225"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.201700 24875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:53.218338 24875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:53.220038 24875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:53.220171 24875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:53.220381 24875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:53.220659 24875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:53.220691 24875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.220721 24875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:53.220745 24875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.226214 24875 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:35221
I20260812 14:31:53.226277 25082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:35221 every 8 connection(s)
I20260812 14:31:53.226559 24875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:31:53.228147 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 24875
I20260812 14:31:53.228245 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:31:53.228415 24247 external_mini_cluster.cc:1428] Setting key d2233ff9ac0653a62d86973a9c3bf957
Aug 12 14:31:53 dist-test-slave-1l3l krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.194@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:53.235803 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.227197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:32805 (local address 127.23.173.254:43215)
0812 14:31:53.227250 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.227252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.227429 (+ 177us) server_negotiation.cc:407] Connection header received
0812 14:31:53.227898 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.227899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.227908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.227957 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:53.228929 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.229522 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.230830 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.231099 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.231158 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.233153 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:53.233155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:53.233155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:53.233163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:53.234433 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.234887 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.234888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.234888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.234935 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.235189 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.235190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.235190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.235280 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:53.235323 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.235652 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.235734 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:31:53.236052 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.227160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:43215 (local address 127.23.173.194:32805)
0812 14:31:53.227315 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.227358 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.227824 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.228017 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.228020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.228651 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:53.228854 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.228859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.229631 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.229632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.230749 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.230752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.231177 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.231178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.231278 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.231766 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:53.231775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:53.233061 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:53.234513 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:53.234522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:53.234524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:53.234815 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:53.234988 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:53.234989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:53.234989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:53.235127 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:53.235331 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:53.235337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:53.235517 (+ 180us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.235645 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.235832 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260812 14:31:53.236531 25083 heartbeater.cc:347] Connected to a master server at 127.23.173.254:43215
I20260812 14:31:53.236618 25083 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:53.236776 25083 heartbeater.cc:511] Master 127.23.173.254:43215 requested a full tablet report, sending...
I20260812 14:31:53.237128 24526 ts_manager.cc:194] Registered new tserver with Master: a7ab2a0e74874cc9a0437cf6730c52fd (127.23.173.194:35221)
I20260812 14:31:53.237640 24526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.194@KRBTEST.COM'} at 127.23.173.194:32805
WARNING: no policy specified for kudu/127.23.173.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
Entry for principal kudu/127.23.173.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab.
I20260812 14:31:53.254173 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:43215
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:53.334688 25092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:53.334885 25092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:53.334916 25092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:53.336084 25092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:53.336125 25092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:53.336160 25092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:53.337579 25092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.195.keytab
--principal=kudu/127.23.173.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:43215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:53.337817 25092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:53.338022 25092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:53.342795 25105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:53.342907 25101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.342989 25092 server_base.cc:1034] running on GCE node
W20260812 14:31:53.342810 25112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.343256 25092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:53.343501 25092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:53.344661 25092 hybrid_clock.cc:648] HybridClock initialized: now 1786545113344645 us; error 28 us; skew 500 ppm
Aug 12 14:31:53 dist-test-slave-1l3l krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:53.347117 25092 init.cc:377] Logged in from keytab as kudu/127.23.173.195@KRBTEST.COM (short username kudu)
I20260812 14:31:53.347517 25092 webserver.cc:460] Webserver started at http://127.23.173.195:33723/ using document root <none> and password file <none>
I20260812 14:31:53.347688 25092 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:53.347757 25092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:53.347858 25092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:53.348805 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "f2f1e53e2e9144208e11e94b7716a59f"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "d4992dad4e893ef9a536ff7a43fba40f"
server_key_iv: "957ac4474cfe98ea50b4d582cbaad35b"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.349102 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "f2f1e53e2e9144208e11e94b7716a59f"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "d4992dad4e893ef9a536ff7a43fba40f"
server_key_iv: "957ac4474cfe98ea50b4d582cbaad35b"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.350268 25092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:53.350885 25121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.351037 25092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:53.351120 25092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "f2f1e53e2e9144208e11e94b7716a59f"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "d4992dad4e893ef9a536ff7a43fba40f"
server_key_iv: "957ac4474cfe98ea50b4d582cbaad35b"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.351181 25092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:53.362681 25092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:53.364441 25092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:53.364574 25092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:53.364775 25092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:53.365061 25092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:53.365095 25092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.365123 25092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:53.365149 25092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.370631 25092 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:43159
I20260812 14:31:53.370687 25326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:43159 every 8 connection(s)
I20260812 14:31:53.370963 25092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:31:53 dist-test-slave-1l3l krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.195@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:31:53.378907 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25092
I20260812 14:31:53.379024 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:31:53.379212 24247 external_mini_cluster.cc:1428] Setting key feb3078764a314d38f1cd55069d18e25
I20260812 14:31:53.379629 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.371627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:42839 (local address 127.23.173.254:43215)
0812 14:31:53.371687 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.371689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.371905 (+ 216us) server_negotiation.cc:407] Connection header received
0812 14:31:53.372384 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.372385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.372399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.372447 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:53.373386 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.373887 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.374836 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.375044 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.375103 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.377139 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:53.377141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:53.377141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:53.377148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:53.378261 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.378605 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.378606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.378606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.378652 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.378889 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.378890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.378890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.379037 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:53.379105 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.379428 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.379513 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:31:53.379860 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.371594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:43215 (local address 127.23.173.195:42839)
0812 14:31:53.371781 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.371818 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.372278 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.372524 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.372528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.373093 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:53.373277 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.373282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.373974 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.373975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.374744 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.374746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.375119 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.375120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.375237 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.375719 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:31:53.375728 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:53.377053 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:53.378329 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:53.378336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:53.378337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:53.378544 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:53.378711 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:31:53.378712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:31:53.378712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:31:53.378828 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:31:53.379131 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:53.379138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:53.379290 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.379389 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.379569 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260812 14:31:53.380380 25328 heartbeater.cc:347] Connected to a master server at 127.23.173.254:43215
I20260812 14:31:53.380465 25328 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:53.380648 25328 heartbeater.cc:511] Master 127.23.173.254:43215 requested a full tablet report, sending...
I20260812 14:31:53.380954 24526 ts_manager.cc:194] Registered new tserver with Master: f2f1e53e2e9144208e11e94b7716a59f (127.23.173.195:43159)
I20260812 14:31:53.381335 24526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.195@KRBTEST.COM'} at 127.23.173.195:42839
I20260812 14:31:53.390925 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:53.397694 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.392191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.23.173.254:43215)
0812 14:31:53.392243 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.392245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.392337 (+ 92us) server_negotiation.cc:407] Connection header received
0812 14:31:53.392418 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.392419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.392429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.392478 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:53.393048 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.393524 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.394236 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.394425 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.394479 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.395587 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:53.395589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:53.395590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:53.395597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:53.396648 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.396942 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.396943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.396944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.396995 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:53.397164 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:53.397165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:53.397165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:53.397314 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:53.397400 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.397522 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.397602 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:31:53.399205 24526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:53.400837 24526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:53.407160 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.402822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:42953 (local address 127.0.0.1:33574)
0812 14:31:53.403273 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.403281 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.403328 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.403761 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.403762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.403765 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:53.403820 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.403822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.405120 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.405121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.406248 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.406251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.406876 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.406877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.406955 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.406968 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.407033 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.407084 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260812 14:31:53.407409 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.403018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.23.173.193:42953)
0812 14:31:53.403249 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.403254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.403345 (+ 91us) server_negotiation.cc:407] Connection header received
0812 14:31:53.403532 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.403535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.403566 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.403647 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:53.403902 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.404952 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.406320 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.406801 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.406867 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.406922 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.407309 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.407353 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":131,"threads_started":1}
I20260812 14:31:53.407433 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.403064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.23.173.195:43159)
0812 14:31:53.403340 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.403344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.403353 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:53.403392 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.403394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.403410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.403502 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:53.403775 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.404874 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.406216 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.406952 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.407026 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.407076 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.407160 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.407198 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260812 14:31:53.407231 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.402822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:43159 (local address 127.0.0.1:56564)
0812 14:31:53.402999 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.403037 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.403094 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.403528 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.403531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.403533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:53.403631 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.403633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.404979 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.404980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.406123 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.406125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.407023 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.407024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.407097 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.407100 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.407150 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.407179 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":31,"thread_start_us":61,"threads_started":1}
I20260812 14:31:53.408181 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 8df78a7ae1fe421799fc8bf6989028dd (DEFAULT_TABLE table=test-table [id=6672c2f931fa4821afca71d1d44fb760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:53.408393 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.403261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.23.173.194:35221)
0812 14:31:53.404604 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.404608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.404619 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:53.404656 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.404658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.404672 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.404745 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:53.404923 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.406068 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.407291 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.408084 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.408146 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.408181 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.408302 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.408334 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":62,"threads_started":1}
I20260812 14:31:53.408573 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df78a7ae1fe421799fc8bf6989028dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:53.411002 25346 tablet_bootstrap.cc:492] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: Bootstrap starting.
I20260812 14:31:53.408391 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.402825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:35221 (local address 127.0.0.1:38166)
0812 14:31:53.403174 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.403182 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.403217 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.404817 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.404818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.404820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:53.404864 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.404865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.406171 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.406171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.407203 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.407206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.408161 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.408163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.408224 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.408228 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.408293 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.408329 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":63,"threads_started":1}
I20260812 14:31:53.411680 25346 tablet_bootstrap.cc:654] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:53.411804 24976 tablet_service.cc:1467] Processing CreateTablet for tablet 8df78a7ae1fe421799fc8bf6989028dd (DEFAULT_TABLE table=test-table [id=6672c2f931fa4821afca71d1d44fb760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:53.411935 25202 tablet_service.cc:1467] Processing CreateTablet for tablet 8df78a7ae1fe421799fc8bf6989028dd (DEFAULT_TABLE table=test-table [id=6672c2f931fa4821afca71d1d44fb760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:53.412133 24976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df78a7ae1fe421799fc8bf6989028dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:53.412284 25202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df78a7ae1fe421799fc8bf6989028dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:53.412307 25346 log.cc:824] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:53.413082 25346 tablet_bootstrap.cc:492] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: No bootstrap required, opened a new log
I20260812 14:31:53.413157 25346 ts_tablet_manager.cc:1397] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:53.414898 25350 tablet_bootstrap.cc:492] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Bootstrap starting.
I20260812 14:31:53.415086 25346 raft_consensus.cc:348] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.415277 25346 raft_consensus.cc:374] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:53.415324 25346 raft_consensus.cc:729] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e4bddf6f424e1e8019482e67a15ea2, State: Initialized, Role: FOLLOWER
I20260812 14:31:53.415498 25346 consensus_queue.cc:260] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.415566 25352 tablet_bootstrap.cc:492] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Bootstrap starting.
I20260812 14:31:53.415655 25350 tablet_bootstrap.cc:654] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:53.415835 25346 ts_tablet_manager.cc:1428] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:53.415977 24868 heartbeater.cc:503] Master 127.23.173.254:43215 was elected leader, sending a full tablet report...
I20260812 14:31:53.416276 25350 log.cc:824] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:53.416591 25352 tablet_bootstrap.cc:654] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:53.417074 25350 tablet_bootstrap.cc:492] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: No bootstrap required, opened a new log
I20260812 14:31:53.417132 25350 ts_tablet_manager.cc:1397] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:53.417164 25352 log.cc:824] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:53.417941 25352 tablet_bootstrap.cc:492] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: No bootstrap required, opened a new log
I20260812 14:31:53.418018 25352 ts_tablet_manager.cc:1397] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:31:53.418601 25350 raft_consensus.cc:348] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.418737 25350 raft_consensus.cc:374] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:53.418761 25350 raft_consensus.cc:729] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ab2a0e74874cc9a0437cf6730c52fd, State: Initialized, Role: FOLLOWER
I20260812 14:31:53.418857 25350 consensus_queue.cc:260] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.419107 25350 ts_tablet_manager.cc:1428] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:31:53.419159 25083 heartbeater.cc:503] Master 127.23.173.254:43215 was elected leader, sending a full tablet report...
I20260812 14:31:53.419874 25352 raft_consensus.cc:348] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.420061 25352 raft_consensus.cc:374] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:53.420110 25352 raft_consensus.cc:729] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f1e53e2e9144208e11e94b7716a59f, State: Initialized, Role: FOLLOWER
I20260812 14:31:53.420217 25352 consensus_queue.cc:260] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.420428 25352 ts_tablet_manager.cc:1428] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:31:53.420485 25328 heartbeater.cc:503] Master 127.23.173.254:43215 was elected leader, sending a full tablet report...
W20260812 14:31:53.477108 25085 tablet.cc:2369] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:31:53.563585 25356 raft_consensus.cc:482] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:53.563760 25356 raft_consensus.cc:504] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.564162 25356 leader_election.cc:288] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34e4bddf6f424e1e8019482e67a15ea2 (127.23.173.193:42953), f2f1e53e2e9144208e11e94b7716a59f (127.23.173.195:43159)
I20260812 14:31:53.568148 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.564522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:48835 (local address 127.23.173.195:43159)
0812 14:31:53.564576 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.564577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.564588 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:53.564623 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.564624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.564632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.564704 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:53.564872 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.565641 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.567166 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.567765 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.567829 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.567838 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.568040 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.568076 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:31:53.568148 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.564524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:59311 (local address 127.23.173.193:42953)
0812 14:31:53.564572 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.564575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.564693 (+ 118us) server_negotiation.cc:407] Connection header received
0812 14:31:53.564746 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.564747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.564757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.564809 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:53.564940 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.565652 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.567166 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.567759 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.567828 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.567838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.568034 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.568076 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:31:53.568159 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.564404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:43159 (local address 127.23.173.194:48835)
0812 14:31:53.564502 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.564515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.564566 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.564721 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.564722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.564724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:53.564803 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.564804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.565738 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.565739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.567067 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.567071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.567862 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.567863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.567959 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.567961 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.568029 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.568069 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":50}
I20260812 14:31:53.568168 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.564404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:42953 (local address 127.23.173.194:59311)
0812 14:31:53.564628 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.564637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.564685 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.564820 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.564820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.564822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:53.564883 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.564885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.565738 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.565739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.567066 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.567071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.567862 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.567863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.567953 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.567959 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.568021 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.568067 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":96,"thread_start_us":142,"threads_started":1}
I20260812 14:31:53.568534 24779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df78a7ae1fe421799fc8bf6989028dd" candidate_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e4bddf6f424e1e8019482e67a15ea2" is_pre_election: true
I20260812 14:31:53.568596 25238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df78a7ae1fe421799fc8bf6989028dd" candidate_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f1e53e2e9144208e11e94b7716a59f" is_pre_election: true
I20260812 14:31:53.568730 25238 raft_consensus.cc:2454] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ab2a0e74874cc9a0437cf6730c52fd in term 0.
I20260812 14:31:53.568730 24779 raft_consensus.cc:2454] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ab2a0e74874cc9a0437cf6730c52fd in term 0.
I20260812 14:31:53.568974 24899 leader_election.cc:302] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e4bddf6f424e1e8019482e67a15ea2, a7ab2a0e74874cc9a0437cf6730c52fd; no voters:
I20260812 14:31:53.569142 25356 raft_consensus.cc:2781] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:53.569201 25356 raft_consensus.cc:482] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:53.569227 25356 raft_consensus.cc:3037] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:53.569979 25356 raft_consensus.cc:504] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.570103 25356 leader_election.cc:288] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [CANDIDATE]: Term 1 election: Requested vote from peers 34e4bddf6f424e1e8019482e67a15ea2 (127.23.173.193:42953), f2f1e53e2e9144208e11e94b7716a59f (127.23.173.195:43159)
I20260812 14:31:53.570272 24779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df78a7ae1fe421799fc8bf6989028dd" candidate_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e4bddf6f424e1e8019482e67a15ea2"
I20260812 14:31:53.570346 24779 raft_consensus.cc:3037] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:53.571064 24779 raft_consensus.cc:2454] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ab2a0e74874cc9a0437cf6730c52fd in term 1.
I20260812 14:31:53.571194 25238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df78a7ae1fe421799fc8bf6989028dd" candidate_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f1e53e2e9144208e11e94b7716a59f"
I20260812 14:31:53.571233 24899 leader_election.cc:302] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e4bddf6f424e1e8019482e67a15ea2, a7ab2a0e74874cc9a0437cf6730c52fd; no voters:
I20260812 14:31:53.571269 25238 raft_consensus.cc:3037] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:53.571321 25356 raft_consensus.cc:2781] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:53.571489 25356 raft_consensus.cc:686] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 1 LEADER]: Becoming Leader. State: Replica: a7ab2a0e74874cc9a0437cf6730c52fd, State: Running, Role: LEADER
I20260812 14:31:53.571585 25356 consensus_queue.cc:237] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } }
I20260812 14:31:53.572108 25238 raft_consensus.cc:2454] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ab2a0e74874cc9a0437cf6730c52fd in term 1.
I20260812 14:31:53.572628 24526 catalog_manager.cc:5495] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd reported cstate change: term changed from 0 to 1, leader changed from <none> to a7ab2a0e74874cc9a0437cf6730c52fd (127.23.173.194). New cstate: current_term: 1 leader_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ab2a0e74874cc9a0437cf6730c52fd" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 35221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 } health_report { overall_health: UNKNOWN } } }
W20260812 14:31:53.573388 24870 tablet.cc:2369] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:31:53.585378 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.583313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.23.173.194:35221)
0812 14:31:53.583356 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.583357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.583366 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:31:53.583431 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.583432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.583447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.583512 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:53.583622 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.584134 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.584820 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.585003 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.585045 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.585151 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:53.585239 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:53.585292 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.585311 (+ 19us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.585327 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:31:53.587023 25238 raft_consensus.cc:1261] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 1 FOLLOWER]: Refusing update from remote peer a7ab2a0e74874cc9a0437cf6730c52fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:53.587023 24779 raft_consensus.cc:1261] T 8df78a7ae1fe421799fc8bf6989028dd P 34e4bddf6f424e1e8019482e67a15ea2 [term 1 FOLLOWER]: Refusing update from remote peer a7ab2a0e74874cc9a0437cf6730c52fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:53.587361 25356 consensus_queue.cc:1035] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e4bddf6f424e1e8019482e67a15ea2" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:53.587504 25363 consensus_queue.cc:1035] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2f1e53e2e9144208e11e94b7716a59f" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:53.589663 25368 mvcc.cc:205] Tried to move back new op lower bound from 7317688785249828864 to 7317688785189744640. Current Snapshot: MvccSnapshot[applied={T|T < 7317688785249828864}]
I20260812 14:31:53.590459 25369 mvcc.cc:205] Tried to move back new op lower bound from 7317688785249828864 to 7317688785189744640. Current Snapshot: MvccSnapshot[applied={T|T < 7317688785249828864}]
I20260812 14:31:53.592067 24525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:53.592154 24525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:53.592876 24525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59fd4feafec464596eeed16fd72a4d6: Sending DeleteTablet for 3 replicas of tablet 8df78a7ae1fe421799fc8bf6989028dd
I20260812 14:31:53.593178 25202 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df78a7ae1fe421799fc8bf6989028dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:53 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:56564
I20260812 14:31:53.593267 24976 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df78a7ae1fe421799fc8bf6989028dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:53 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:38166
I20260812 14:31:53.593400 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df78a7ae1fe421799fc8bf6989028dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:53 UTC) from {username='kudu', principal='kudu/127.23.173.254@KRBTEST.COM'} at 127.0.0.1:33574
I20260812 14:31:53.593497 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24637
I20260812 14:31:53.593616 25387 tablet_replica.cc:330] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: stopping tablet replica
I20260812 14:31:53.593709 25387 raft_consensus.cc:2229] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:31:53.593788 25387 raft_consensus.cc:2258] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:53.596755 25387 ts_tablet_manager.cc:1905] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:53.598688 25387 ts_tablet_manager.cc:1918] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:53.598785 25387 log.cc:1196] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/8df78a7ae1fe421799fc8bf6989028dd
I20260812 14:31:53.599102 25387 ts_tablet_manager.cc:1939] T 8df78a7ae1fe421799fc8bf6989028dd P a7ab2a0e74874cc9a0437cf6730c52fd: Deleting consensus metadata
I20260812 14:31:53.599566 24503 catalog_manager.cc:4841] TS a7ab2a0e74874cc9a0437cf6730c52fd (127.23.173.194:35221): tablet 8df78a7ae1fe421799fc8bf6989028dd (table test-table [id=6672c2f931fa4821afca71d1d44fb760]) successfully deleted
I20260812 14:31:53.600512 25385 tablet_replica.cc:330] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: stopping tablet replica
I20260812 14:31:53.601754 25385 raft_consensus.cc:2229] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:53.601948 25385 raft_consensus.cc:2258] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:53.603411 25385 ts_tablet_manager.cc:1905] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:53.605087 25385 ts_tablet_manager.cc:1918] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:53.605217 25385 log.cc:1196] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/8df78a7ae1fe421799fc8bf6989028dd
I20260812 14:31:53.605584 25385 ts_tablet_manager.cc:1939] T 8df78a7ae1fe421799fc8bf6989028dd P f2f1e53e2e9144208e11e94b7716a59f: Deleting consensus metadata
I20260812 14:31:53.606035 24501 catalog_manager.cc:4841] TS f2f1e53e2e9144208e11e94b7716a59f (127.23.173.195:43159): tablet 8df78a7ae1fe421799fc8bf6989028dd (table test-table [id=6672c2f931fa4821afca71d1d44fb760]) successfully deleted
W20260812 14:31:53.608350 24498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:42953 (error 108)
I20260812 14:31:53.608856 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.608725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:42953 (local address 127.0.0.1:33576)
0812 14:31:53.608780 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.608821 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 14:31:53.608887 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24875
W20260812 14:31:53.612190 24498 catalog_manager.cc:4584] TS 34e4bddf6f424e1e8019482e67a15ea2 (127.23.173.193:42953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8df78a7ae1fe421799fc8bf6989028dd: Network error: Client connection negotiation failed: client connection to 127.23.173.193:42953: connect: Connection refused (error 111)
I20260812 14:31:53.618661 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25092
I20260812 14:31:53.629213 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 24467
2026-08-12T14:31:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2188 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T14:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:53Z Disabled control of system clock
I20260812 14:31:53.663101 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46361
--webserver_interface=127.23.173.254
--webserver_port=0
--builtin_ntp_servers=127.23.173.212:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:53.743856 25400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:53.744076 25400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:53.744096 25400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:53.745236 25400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:53.745277 25400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:53.745289 25400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:53.745312 25400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:53.745332 25400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:53.746655 25400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:53.746876 25400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:53.747124 25400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:53.751667 25410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.751722 25400 server_base.cc:1034] running on GCE node
W20260812 14:31:53.751664 25406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:53.751664 25407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.752131 25400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:53.752409 25400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:53.753566 25400 hybrid_clock.cc:648] HybridClock initialized: now 1786545113753547 us; error 33 us; skew 500 ppm
I20260812 14:31:53.754772 25400 webserver.cc:460] Webserver started at http://127.23.173.254:36475/ using document root <none> and password file <none>
I20260812 14:31:53.755033 25400 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:53.755092 25400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:53.755229 25400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:53.756364 25400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "54ea547096f44eb9972460cbb55126f8"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "a3c368173a2fc93994684d378cc6b7ae"
server_key_iv: "6c5ed39fe4aed9dd3cf1e597d82cf507"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.756711 25400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "54ea547096f44eb9972460cbb55126f8"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "a3c368173a2fc93994684d378cc6b7ae"
server_key_iv: "6c5ed39fe4aed9dd3cf1e597d82cf507"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.757920 25400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:53.758531 25422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.758683 25400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:53.758764 25400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "54ea547096f44eb9972460cbb55126f8"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "a3c368173a2fc93994684d378cc6b7ae"
server_key_iv: "6c5ed39fe4aed9dd3cf1e597d82cf507"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.758831 25400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:53.776240 25400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:53.776521 25400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:53.776660 25400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:53.780453 25400 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46361
I20260812 14:31:53.780491 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46361 every 8 connection(s)
I20260812 14:31:53.780802 25400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:53.781735 25509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:53.784020 25509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8: Bootstrap starting.
I20260812 14:31:53.784560 25509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:53.784798 25509 log.cc:824] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:53.785379 25509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8: No bootstrap required, opened a new log
I20260812 14:31:53.786933 25509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ea547096f44eb9972460cbb55126f8" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46361 } }
I20260812 14:31:53.787067 25509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:53.787089 25509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ea547096f44eb9972460cbb55126f8, State: Initialized, Role: FOLLOWER
I20260812 14:31:53.787200 25509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ea547096f44eb9972460cbb55126f8" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46361 } }
I20260812 14:31:53.787279 25509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:53.787317 25509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:53.787366 25509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:53.787391 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25400
I20260812 14:31:53.787487 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:53.787675 24247 external_mini_cluster.cc:1428] Setting key 89e9423d1005e313be42671da6ec9d84
I20260812 14:31:53.788034 25509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ea547096f44eb9972460cbb55126f8" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46361 } }
I20260812 14:31:53.788141 25509 leader_election.cc:302] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54ea547096f44eb9972460cbb55126f8; no voters:
I20260812 14:31:53.788321 25509 leader_election.cc:288] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:53.788368 25514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:53.788514 25514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [term 1 LEADER]: Becoming Leader. State: Replica: 54ea547096f44eb9972460cbb55126f8, State: Running, Role: LEADER
I20260812 14:31:53.788591 25509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:53.788661 25514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ea547096f44eb9972460cbb55126f8" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46361 } }
I20260812 14:31:53.789124 25521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54ea547096f44eb9972460cbb55126f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ea547096f44eb9972460cbb55126f8" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46361 } } }
I20260812 14:31:53.789216 25521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:53.789420 25524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54ea547096f44eb9972460cbb55126f8. Latest consensus state: current_term: 1 leader_uuid: "54ea547096f44eb9972460cbb55126f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ea547096f44eb9972460cbb55126f8" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46361 } } }
I20260812 14:31:53.789705 25524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:53.789902 25534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:53.790366 25534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:53.791896 25534 catalog_manager.cc:1347] Generated new cluster ID: 86242b3e22174370813658b3b4406f2c
I20260812 14:31:53.791949 25534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:53.793694 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.788190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.23.173.254:46361)
0812 14:31:53.788423 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.788432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.788464 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:31:53.788578 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.788585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.788649 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.788780 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:53.788994 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.790685 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.791579 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.792178 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.792297 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.792991 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:53.792995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:53.792996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:53.793211 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:53.793254 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:53.793269 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:53.793339 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.793396 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.793478 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260812 14:31:53.799613 25534 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:53.800304 25534 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:53.817553 25534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8: Generated new TSK 0
I20260812 14:31:53.817792 25534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:31:53.825125 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.193:0
--local_ip_for_outbound_sockets=127.23.173.193
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46361
--builtin_ntp_servers=127.23.173.212:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:53.909776 25552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:53.909991 25552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:53.910023 25552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:53.911186 25552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:53.911229 25552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:53.911262 25552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.193
I20260812 14:31:53.912698 25552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.193
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.193
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:53.912943 25552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:53.913239 25552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:53.919116 25557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:53.919116 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.919375 25552 server_base.cc:1034] running on GCE node
W20260812 14:31:53.919183 25559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:53.919637 25552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:53.919917 25552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:53.921066 25552 hybrid_clock.cc:648] HybridClock initialized: now 1786545113921054 us; error 30 us; skew 500 ppm
I20260812 14:31:53.922399 25552 webserver.cc:460] Webserver started at http://127.23.173.193:46795/ using document root <none> and password file <none>
I20260812 14:31:53.922662 25552 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:53.922714 25552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:53.922847 25552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:53.923969 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data/instance:
uuid: "2713ab268f21467c8643bd905ee147d7"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "666b07b0f8166a2fd1f8328e60ddfcc0"
server_key_iv: "5c2928465f0aeda9656864cf0fd977ed"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.924309 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance:
uuid: "2713ab268f21467c8643bd905ee147d7"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "666b07b0f8166a2fd1f8328e60ddfcc0"
server_key_iv: "5c2928465f0aeda9656864cf0fd977ed"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.925623 25552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:53.926328 25572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.926463 25552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:53.926537 25552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal
uuid: "2713ab268f21467c8643bd905ee147d7"
format_stamp: "Formatted at 2026-08-12 14:31:53 on dist-test-slave-1l3l"
server_key: "666b07b0f8166a2fd1f8328e60ddfcc0"
server_key_iv: "5c2928465f0aeda9656864cf0fd977ed"
server_key_version: "encryptionkey@0"
I20260812 14:31:53.926616 25552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:53.951077 25552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:53.951390 25552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:53.951522 25552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:53.951808 25552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:53.952162 25552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:53.952200 25552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.952230 25552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:53.952248 25552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:53.957804 25552 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.193:41241
I20260812 14:31:53.957882 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.193:41241 every 8 connection(s)
I20260812 14:31:53.958174 25552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/data/info.pb
I20260812 14:31:53.961265 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25552
I20260812 14:31:53.961364 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-0/wal/instance
I20260812 14:31:53.961547 24247 external_mini_cluster.cc:1428] Setting key 4c412d9ad23c4005fbd218a44af7d6ea
I20260812 14:31:53.962451 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.194:0
--local_ip_for_outbound_sockets=127.23.173.194
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46361
--builtin_ntp_servers=127.23.173.212:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:31:53.963490 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.958868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.193:51715 (local address 127.23.173.254:46361)
0812 14:31:53.958929 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:53.958931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:53.959189 (+ 258us) server_negotiation.cc:407] Connection header received
0812 14:31:53.959723 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:53.959725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:53.959740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:53.959796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:53.960475 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.960986 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.962113 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.962307 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.962377 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.963228 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:53.963230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:53.963230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:53.963235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:53.963252 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:53.963256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:53.963299 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:53.963378 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:31:53.963411 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:31:53.963670 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:53.958864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46361 (local address 127.23.173.193:51715)
0812 14:31:53.959071 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:53.959107 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:53.959603 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:53.959875 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:53.959879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:53.959887 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:53.960384 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.960388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.961074 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.961076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.962020 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:53.962023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:53.962396 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:53.962397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:53.962550 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:53.963100 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:53.963100 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:53.963129 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:53.963151 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:53.963306 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:53.963311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:53.963314 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:53.963372 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:31:53.963470 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260812 14:31:53.964440 25773 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46361
I20260812 14:31:53.964526 25773 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:53.964707 25773 heartbeater.cc:511] Master 127.23.173.254:46361 requested a full tablet report, sending...
I20260812 14:31:53.965184 25449 ts_manager.cc:194] Registered new tserver with Master: 2713ab268f21467c8643bd905ee147d7 (127.23.173.193:41241)
I20260812 14:31:53.965788 25449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.193:51715
W20260812 14:31:54.044999 25776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:54.045214 25776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:54.045244 25776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:54.046376 25776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:54.046415 25776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:54.046447 25776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.194
I20260812 14:31:54.047818 25776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.194
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.194
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:54.048062 25776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:54.048349 25776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:54.053144 25786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:54.053201 25795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:54.053151 25791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:54.053382 25776 server_base.cc:1034] running on GCE node
I20260812 14:31:54.053601 25776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:54.053831 25776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:54.054970 25776 hybrid_clock.cc:648] HybridClock initialized: now 1786545114054955 us; error 27 us; skew 500 ppm
I20260812 14:31:54.056289 25776 webserver.cc:460] Webserver started at http://127.23.173.194:45073/ using document root <none> and password file <none>
I20260812 14:31:54.056529 25776 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:54.056576 25776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:54.056704 25776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:54.057770 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data/instance:
uuid: "a0b45650fb974387963312e024ef1e9d"
format_stamp: "Formatted at 2026-08-12 14:31:54 on dist-test-slave-1l3l"
server_key: "aa89a4eaea4af48c4839529b570fc651"
server_key_iv: "486f18a0befdfe68e83783daa26d72d5"
server_key_version: "encryptionkey@0"
I20260812 14:31:54.058115 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance:
uuid: "a0b45650fb974387963312e024ef1e9d"
format_stamp: "Formatted at 2026-08-12 14:31:54 on dist-test-slave-1l3l"
server_key: "aa89a4eaea4af48c4839529b570fc651"
server_key_iv: "486f18a0befdfe68e83783daa26d72d5"
server_key_version: "encryptionkey@0"
I20260812 14:31:54.059343 25776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:54.060096 25808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.060262 25776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.060343 25776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal
uuid: "a0b45650fb974387963312e024ef1e9d"
format_stamp: "Formatted at 2026-08-12 14:31:54 on dist-test-slave-1l3l"
server_key: "aa89a4eaea4af48c4839529b570fc651"
server_key_iv: "486f18a0befdfe68e83783daa26d72d5"
server_key_version: "encryptionkey@0"
I20260812 14:31:54.060398 25776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:54.073694 25776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:54.073983 25776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:54.074095 25776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:54.074286 25776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:54.074585 25776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:54.074617 25776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.074645 25776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:54.074661 25776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.080255 25776 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.194:43835
I20260812 14:31:54.080330 25985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.194:43835 every 8 connection(s)
I20260812 14:31:54.080643 25776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/data/info.pb
I20260812 14:31:54.085839 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.081384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.194:42737 (local address 127.23.173.254:46361)
0812 14:31:54.081443 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.081445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.081697 (+ 252us) server_negotiation.cc:407] Connection header received
0812 14:31:54.082210 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.082211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.082224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.082280 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:54.083020 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.083509 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.084508 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.084710 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.084769 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.085575 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:54.085576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:54.085577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:54.085582 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:54.085597 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:54.085601 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:54.085645 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.085722 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.085753 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:31:54.086024 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.081373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46361 (local address 127.23.173.194:42737)
0812 14:31:54.081574 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.081611 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.082105 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.082399 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.082403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.082411 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:54.082924 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.082929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.083603 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.083604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.084423 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.084425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.084788 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.084789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.084925 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.085435 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:54.085436 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:54.085464 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:54.085488 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:54.085651 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:54.085657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:54.085660 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.085720 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.085826 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260812 14:31:54.086520 25986 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46361
I20260812 14:31:54.086614 25986 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:54.086800 25986 heartbeater.cc:511] Master 127.23.173.254:46361 requested a full tablet report, sending...
I20260812 14:31:54.087162 25449 ts_manager.cc:194] Registered new tserver with Master: a0b45650fb974387963312e024ef1e9d (127.23.173.194:43835)
I20260812 14:31:54.087532 25449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.194:42737
I20260812 14:31:54.088184 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25776
I20260812 14:31:54.088304 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal/instance
I20260812 14:31:54.088503 24247 external_mini_cluster.cc:1428] Setting key 80a38ec0c060dea6621378b17d25ec7b
I20260812 14:31:54.089386 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.195:0
--local_ip_for_outbound_sockets=127.23.173.195
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46361
--builtin_ntp_servers=127.23.173.212:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:31:54.172113 25993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:54.172353 25993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:54.172374 25993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:54.173569 25993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:54.173615 25993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:54.173640 25993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.195
I20260812 14:31:54.174978 25993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.195
--webserver_port=0
--tserver_master_addrs=127.23.173.254:46361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.195
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:54.175191 25993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:54.175482 25993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:54.180743 26011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:54.180851 25993 server_base.cc:1034] running on GCE node
W20260812 14:31:54.180755 26013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:54.180755 26007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:54.181154 25993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:54.181394 25993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:54.182539 25993 hybrid_clock.cc:648] HybridClock initialized: now 1786545114182521 us; error 36 us; skew 500 ppm
I20260812 14:31:54.183856 25993 webserver.cc:460] Webserver started at http://127.23.173.195:32887/ using document root <none> and password file <none>
I20260812 14:31:54.184126 25993 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:54.184180 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:54.184314 25993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:54.185389 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data/instance:
uuid: "d427f698d17d484cbead43b1e02052d7"
format_stamp: "Formatted at 2026-08-12 14:31:54 on dist-test-slave-1l3l"
server_key: "ec06bea27554967e6bd3f132bd9ec681"
server_key_iv: "163e2a8656d2ae26448952831bf9b8d6"
server_key_version: "encryptionkey@0"
I20260812 14:31:54.185730 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance:
uuid: "d427f698d17d484cbead43b1e02052d7"
format_stamp: "Formatted at 2026-08-12 14:31:54 on dist-test-slave-1l3l"
server_key: "ec06bea27554967e6bd3f132bd9ec681"
server_key_iv: "163e2a8656d2ae26448952831bf9b8d6"
server_key_version: "encryptionkey@0"
I20260812 14:31:54.187034 25993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:31:54.187805 26024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.187955 25993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:31:54.188040 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal
uuid: "d427f698d17d484cbead43b1e02052d7"
format_stamp: "Formatted at 2026-08-12 14:31:54 on dist-test-slave-1l3l"
server_key: "ec06bea27554967e6bd3f132bd9ec681"
server_key_iv: "163e2a8656d2ae26448952831bf9b8d6"
server_key_version: "encryptionkey@0"
I20260812 14:31:54.188122 25993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:54.207448 25993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:54.207805 25993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:54.207957 25993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:54.208216 25993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:31:54.208554 25993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:31:54.208590 25993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.208621 25993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:31:54.208637 25993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:54.214313 25993 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.195:36375
I20260812 14:31:54.214368 26228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.195:36375 every 8 connection(s)
I20260812 14:31:54.214696 25993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/data/info.pb
I20260812 14:31:54.219731 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.215437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:42609 (local address 127.23.173.254:46361)
0812 14:31:54.215508 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.215510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.215748 (+ 238us) server_negotiation.cc:407] Connection header received
0812 14:31:54.216261 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.216262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.216271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.216338 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:54.216972 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.217464 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.218422 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.218633 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.218688 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.219458 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:54.219459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:54.219459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:54.219482 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:54.219499 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:54.219503 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:54.219549 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.219619 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.219647 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:31:54.219937 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.215406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46361 (local address 127.23.173.195:42609)
0812 14:31:54.215605 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.215647 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.216176 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.216395 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.216399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.216407 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:31:54.216906 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.216911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.217541 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.217542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.218337 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.218340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.218702 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.218703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.218820 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.219324 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:31:54.219324 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:31:54.219354 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:31:54.219378 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:31:54.219556 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:31:54.219559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:31:54.219562 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.219611 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.219695 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260812 14:31:54.220398 26229 heartbeater.cc:347] Connected to a master server at 127.23.173.254:46361
I20260812 14:31:54.220499 26229 heartbeater.cc:464] Registering TS with master...
I20260812 14:31:54.220688 26229 heartbeater.cc:511] Master 127.23.173.254:46361 requested a full tablet report, sending...
I20260812 14:31:54.221051 25449 ts_manager.cc:194] Registered new tserver with Master: d427f698d17d484cbead43b1e02052d7 (127.23.173.195:36375)
I20260812 14:31:54.221400 25449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.195:42609
I20260812 14:31:54.224807 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 25993
I20260812 14:31:54.224896 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal/instance
I20260812 14:31:54.225095 24247 external_mini_cluster.cc:1428] Setting key c62c94885f7ebc5441f9db1897b4ecab
I20260812 14:31:54.226266 24247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:31:54.230758 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.227815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.23.173.254:46361)
0812 14:31:54.227863 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.227865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.227997 (+ 132us) server_negotiation.cc:407] Connection header received
0812 14:31:54.228064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.228065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.228080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.228122 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:54.228286 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.228813 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.229583 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.229764 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.229819 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.230494 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:54.230495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:54.230496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:54.230500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:54.230517 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:54.230522 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:54.230583 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.230640 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.230670 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:31:54.232676 25449 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:31:54.234261 25449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:31:54.240691 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.236509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.195:36375 (local address 127.0.0.1:56228)
0812 14:31:54.236832 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.236866 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.236925 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.237211 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.237214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.237216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:54.237288 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.237291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.238668 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.238669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.239747 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.239749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.240401 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.240402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.240481 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.240496 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.240555 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.240609 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":32,"thread_start_us":117,"threads_started":1}
I20260812 14:31:54.240701 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.236785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.23.173.195:36375)
0812 14:31:54.236980 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.236985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.236996 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:31:54.237042 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.237044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.237063 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.237137 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:54.237360 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.238571 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.239810 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.240308 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.240382 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.240489 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.240581 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.240620 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260812 14:31:54.240845 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.236510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:43835 (local address 127.0.0.1:48678)
0812 14:31:54.237036 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.237045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.237095 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.237251 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.237252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.237253 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:54.237288 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.237290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.238686 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.238688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.239560 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.239563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.240599 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.240600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.240654 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.240660 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.240709 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.240753 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":19,"thread_start_us":115,"threads_started":1}
I20260812 14:31:54.241077 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.236795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.23.173.194:43835)
0812 14:31:54.237009 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.237013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.237107 (+ 94us) server_negotiation.cc:407] Connection header received
0812 14:31:54.237149 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.237151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.237164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.237425 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:54.237490 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.238601 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.239648 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.240383 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.240456 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.240569 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.240958 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.241000 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260812 14:31:54.241106 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.236614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.23.173.193:41241)
0812 14:31:54.237092 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.237096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.237106 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:31:54.237146 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.237149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.237163 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.237247 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:54.237462 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.238200 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.239591 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.240390 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.240456 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.240545 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.240974 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.241032 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":114,"threads_started":1}
I20260812 14:31:54.241247 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.236503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:41241 (local address 127.0.0.1:53992)
0812 14:31:54.236807 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.236855 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.236950 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.237354 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.237355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.237356 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:54.237382 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.237384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.238281 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.238283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.239513 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.239516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.240584 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.240585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.240654 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.240662 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.240709 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.241182 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260812 14:31:54.241559 26117 tablet_service.cc:1467] Processing CreateTablet for tablet 6d838f07f1224832aeffd37f9964130e (DEFAULT_TABLE table=test-table [id=010a1f9ed2cc48ceab53ab53454f5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:54.241986 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d838f07f1224832aeffd37f9964130e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:54.241907 25655 tablet_service.cc:1467] Processing CreateTablet for tablet 6d838f07f1224832aeffd37f9964130e (DEFAULT_TABLE table=test-table [id=010a1f9ed2cc48ceab53ab53454f5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:54.241907 25886 tablet_service.cc:1467] Processing CreateTablet for tablet 6d838f07f1224832aeffd37f9964130e (DEFAULT_TABLE table=test-table [id=010a1f9ed2cc48ceab53ab53454f5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:31:54.242237 25886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d838f07f1224832aeffd37f9964130e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:54.242237 25655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d838f07f1224832aeffd37f9964130e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:54.244884 26245 tablet_bootstrap.cc:492] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Bootstrap starting.
I20260812 14:31:54.245498 26245 tablet_bootstrap.cc:654] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:54.245548 26248 tablet_bootstrap.cc:492] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Bootstrap starting.
I20260812 14:31:54.245836 26245 log.cc:824] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:54.246194 26248 tablet_bootstrap.cc:654] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:54.246498 26248 log.cc:824] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:54.246637 26245 tablet_bootstrap.cc:492] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: No bootstrap required, opened a new log
I20260812 14:31:54.246711 26245 ts_tablet_manager.cc:1397] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:54.246876 26247 tablet_bootstrap.cc:492] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: Bootstrap starting.
I20260812 14:31:54.247222 26248 tablet_bootstrap.cc:492] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: No bootstrap required, opened a new log
I20260812 14:31:54.247285 26248 ts_tablet_manager.cc:1397] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:31:54.247546 26247 tablet_bootstrap.cc:654] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:54.248031 26247 log.cc:824] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:54.248911 26247 tablet_bootstrap.cc:492] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: No bootstrap required, opened a new log
I20260812 14:31:54.248986 26247 ts_tablet_manager.cc:1397] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:31:54.249258 26245 raft_consensus.cc:348] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.249434 26245 raft_consensus.cc:374] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:54.249471 26245 raft_consensus.cc:729] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d427f698d17d484cbead43b1e02052d7, State: Initialized, Role: FOLLOWER
I20260812 14:31:54.249442 26248 raft_consensus.cc:348] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.249626 26245 consensus_queue.cc:260] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.249799 26248 raft_consensus.cc:374] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:54.249851 26248 raft_consensus.cc:729] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b45650fb974387963312e024ef1e9d, State: Initialized, Role: FOLLOWER
I20260812 14:31:54.249890 26245 ts_tablet_manager.cc:1428] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:54.249991 26248 consensus_queue.cc:260] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.250046 26229 heartbeater.cc:503] Master 127.23.173.254:46361 was elected leader, sending a full tablet report...
I20260812 14:31:54.250254 26248 ts_tablet_manager.cc:1428] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:31:54.250367 25986 heartbeater.cc:503] Master 127.23.173.254:46361 was elected leader, sending a full tablet report...
I20260812 14:31:54.251390 26247 raft_consensus.cc:348] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.251539 26247 raft_consensus.cc:374] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:54.251561 26247 raft_consensus.cc:729] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2713ab268f21467c8643bd905ee147d7, State: Initialized, Role: FOLLOWER
I20260812 14:31:54.251645 26247 consensus_queue.cc:260] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.251888 26247 ts_tablet_manager.cc:1428] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:31:54.251966 25773 heartbeater.cc:503] Master 127.23.173.254:46361 was elected leader, sending a full tablet report...
I20260812 14:31:54.274026 26256 raft_consensus.cc:482] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:31:54.274168 26256 raft_consensus.cc:504] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.274703 26256 leader_election.cc:288] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2713ab268f21467c8643bd905ee147d7 (127.23.173.193:41241), a0b45650fb974387963312e024ef1e9d (127.23.173.194:43835)
I20260812 14:31:54.277951 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.274992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:43057 (local address 127.23.173.194:43835)
0812 14:31:54.275052 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.275055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.275068 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:31:54.275140 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.275141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.275149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.275213 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:54.275413 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.276003 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.277229 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.277602 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.277661 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.277669 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.277821 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.277870 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:31:54.277952 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.274921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.194:43835 (local address 127.23.173.195:43057)
0812 14:31:54.275012 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.275025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.275074 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.275231 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.275232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.275234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:54.275310 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.275313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.276080 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.276081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.277116 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.277119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.277677 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.277678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.277750 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.277754 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.277797 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.277841 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":152}
I20260812 14:31:54.277956 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.274995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.195:37571 (local address 127.23.173.193:41241)
0812 14:31:54.275058 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.275060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.275274 (+ 214us) server_negotiation.cc:407] Connection header received
0812 14:31:54.275320 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.275321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.275334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.275398 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:31:54.275579 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.276120 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.277219 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.277632 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.277706 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.277714 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.277851 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.277894 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:31:54.277976 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.274921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:41241 (local address 127.23.173.195:37571)
0812 14:31:54.275223 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.275230 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:31:54.275275 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:31:54.275420 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:31:54.275421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:31:54.275423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:31:54.275503 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.275505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.276232 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.276233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.277115 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:31:54.277118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.277722 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.277723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:31:54.277779 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.277782 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:31:54.277837 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:31:54.277872 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":100,"thread_start_us":136,"threads_started":1}
I20260812 14:31:54.278317 25688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d838f07f1224832aeffd37f9964130e" candidate_uuid: "d427f698d17d484cbead43b1e02052d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2713ab268f21467c8643bd905ee147d7" is_pre_election: true
I20260812 14:31:54.278326 25922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d838f07f1224832aeffd37f9964130e" candidate_uuid: "d427f698d17d484cbead43b1e02052d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b45650fb974387963312e024ef1e9d" is_pre_election: true
I20260812 14:31:54.278484 25688 raft_consensus.cc:2454] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d427f698d17d484cbead43b1e02052d7 in term 0.
I20260812 14:31:54.278484 25922 raft_consensus.cc:2454] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d427f698d17d484cbead43b1e02052d7 in term 0.
I20260812 14:31:54.278715 26029 leader_election.cc:302] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b45650fb974387963312e024ef1e9d, d427f698d17d484cbead43b1e02052d7; no voters:
I20260812 14:31:54.278891 26256 raft_consensus.cc:2781] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:31:54.278990 26256 raft_consensus.cc:482] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:31:54.279014 26256 raft_consensus.cc:3037] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:54.279888 26256 raft_consensus.cc:504] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.280050 26256 leader_election.cc:288] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [CANDIDATE]: Term 1 election: Requested vote from peers 2713ab268f21467c8643bd905ee147d7 (127.23.173.193:41241), a0b45650fb974387963312e024ef1e9d (127.23.173.194:43835)
I20260812 14:31:54.280252 25688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d838f07f1224832aeffd37f9964130e" candidate_uuid: "d427f698d17d484cbead43b1e02052d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2713ab268f21467c8643bd905ee147d7"
I20260812 14:31:54.280337 25688 raft_consensus.cc:3037] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:54.281046 25688 raft_consensus.cc:2454] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d427f698d17d484cbead43b1e02052d7 in term 1.
I20260812 14:31:54.281168 25922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d838f07f1224832aeffd37f9964130e" candidate_uuid: "d427f698d17d484cbead43b1e02052d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b45650fb974387963312e024ef1e9d"
I20260812 14:31:54.281243 26025 leader_election.cc:302] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2713ab268f21467c8643bd905ee147d7, d427f698d17d484cbead43b1e02052d7; no voters:
I20260812 14:31:54.281265 25922 raft_consensus.cc:3037] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:54.281339 26256 raft_consensus.cc:2781] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:54.281525 26256 raft_consensus.cc:686] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 1 LEADER]: Becoming Leader. State: Replica: d427f698d17d484cbead43b1e02052d7, State: Running, Role: LEADER
I20260812 14:31:54.281670 26256 consensus_queue.cc:237] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } }
I20260812 14:31:54.282235 25922 raft_consensus.cc:2454] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d427f698d17d484cbead43b1e02052d7 in term 1.
I20260812 14:31:54.282507 25449 catalog_manager.cc:5495] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d427f698d17d484cbead43b1e02052d7 (127.23.173.195). New cstate: current_term: 1 leader_uuid: "d427f698d17d484cbead43b1e02052d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d427f698d17d484cbead43b1e02052d7" member_type: VOTER last_known_addr { host: "127.23.173.195" port: 36375 } health_report { overall_health: HEALTHY } } }
I20260812 14:31:54.298410 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.296280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.23.173.195:36375)
0812 14:31:54.296332 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:54.296334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:54.296341 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:31:54.296405 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:54.296405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:54.296424 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:54.296486 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:31:54.296646 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.297131 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.297831 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:54.298014 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:54.298060 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:54.298165 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:31:54.298265 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:31:54.298321 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:54.298334 (+ 13us) server_negotiation.cc:299] Negotiation successful
0812 14:31:54.298350 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:31:54.300282 25688 raft_consensus.cc:1261] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 1 FOLLOWER]: Refusing update from remote peer d427f698d17d484cbead43b1e02052d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:54.300282 25922 raft_consensus.cc:1261] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 1 FOLLOWER]: Refusing update from remote peer d427f698d17d484cbead43b1e02052d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:31:54.300684 26262 consensus_queue.cc:1035] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2713ab268f21467c8643bd905ee147d7" member_type: VOTER last_known_addr { host: "127.23.173.193" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:54.300822 26256 consensus_queue.cc:1035] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b45650fb974387963312e024ef1e9d" member_type: VOTER last_known_addr { host: "127.23.173.194" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:31:54.304051 26270 mvcc.cc:205] Tried to move back new op lower bound from 7317688788171055104 to 7317688788098424832. Current Snapshot: MvccSnapshot[applied={T|T < 7317688788171055104}]
I20260812 14:31:54.304244 26268 mvcc.cc:205] Tried to move back new op lower bound from 7317688788171055104 to 7317688788098424832. Current Snapshot: MvccSnapshot[applied={T|T < 7317688788171055104 or (T in {7317688788171055104})}]
I20260812 14:31:54.305862 25448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:54.305958 25448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:31:54.306653 25448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54ea547096f44eb9972460cbb55126f8: Sending DeleteTablet for 3 replicas of tablet 6d838f07f1224832aeffd37f9964130e
I20260812 14:31:54.307087 25886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d838f07f1224832aeffd37f9964130e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:54 UTC) from {username='slave'} at 127.0.0.1:48678
I20260812 14:31:54.307138 25655 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d838f07f1224832aeffd37f9964130e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:54 UTC) from {username='slave'} at 127.0.0.1:53992
I20260812 14:31:54.307137 26117 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d838f07f1224832aeffd37f9964130e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:31:54 UTC) from {username='slave'} at 127.0.0.1:56228
I20260812 14:31:54.307359 26281 tablet_replica.cc:330] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7: stopping tablet replica
I20260812 14:31:54.307466 26281 raft_consensus.cc:2229] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:54.307585 26281 raft_consensus.cc:2258] T 6d838f07f1224832aeffd37f9964130e P 2713ab268f21467c8643bd905ee147d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:54.307693 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25552
I20260812 14:31:54.310498 26282 tablet_replica.cc:330] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: stopping tablet replica
I20260812 14:31:54.310621 26282 raft_consensus.cc:2229] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:31:54.310719 26282 raft_consensus.cc:2258] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:54.310853 26280 tablet_replica.cc:330] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: stopping tablet replica
I20260812 14:31:54.310954 26280 raft_consensus.cc:2229] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:31:54.311050 26280 raft_consensus.cc:2258] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:31:54.311447 26282 ts_tablet_manager.cc:1905] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:54.311512 26280 ts_tablet_manager.cc:1905] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:31:54.313500 26282 ts_tablet_manager.cc:1918] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:54.313525 26280 ts_tablet_manager.cc:1918] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:31:54.313616 26280 log.cc:1196] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-1/wal/wals/6d838f07f1224832aeffd37f9964130e
I20260812 14:31:54.313752 26282 log.cc:1196] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Deleting WAL directory at /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544840008271-24247-0/minicluster-data/ts-2/wal/wals/6d838f07f1224832aeffd37f9964130e
I20260812 14:31:54.313912 26280 ts_tablet_manager.cc:1939] T 6d838f07f1224832aeffd37f9964130e P a0b45650fb974387963312e024ef1e9d: Deleting consensus metadata
I20260812 14:31:54.314144 26282 ts_tablet_manager.cc:1939] T 6d838f07f1224832aeffd37f9964130e P d427f698d17d484cbead43b1e02052d7: Deleting consensus metadata
I20260812 14:31:54.314383 25428 catalog_manager.cc:4841] TS a0b45650fb974387963312e024ef1e9d (127.23.173.194:43835): tablet 6d838f07f1224832aeffd37f9964130e (table test-table [id=010a1f9ed2cc48ceab53ab53454f5be8]) successfully deleted
I20260812 14:31:54.314690 25426 catalog_manager.cc:4841] TS d427f698d17d484cbead43b1e02052d7 (127.23.173.195:36375): tablet 6d838f07f1224832aeffd37f9964130e (table test-table [id=010a1f9ed2cc48ceab53ab53454f5be8]) successfully deleted
W20260812 14:31:54.322497 25425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.193:41241 (error 108)
I20260812 14:31:54.322674 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25776
I20260812 14:31:54.328258 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:54.328078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.193:41241 (local address 127.0.0.1:53998)
0812 14:31:54.328145 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 14:31:54.328207 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.193:41241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 14:31:54.328433 25425 catalog_manager.cc:4584] TS 2713ab268f21467c8643bd905ee147d7 (127.23.173.193:41241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d838f07f1224832aeffd37f9964130e: Network error: Client connection negotiation failed: client connection to 127.23.173.193:41241: connect: Connection refused (error 111)
I20260812 14:31:54.336999 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25993
I20260812 14:31:54.349749 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 25400
2026-08-12T14:31:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (721 ms)
[----------] 6 tests from EncryptionPolicyTest (9064 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:54 dist-test-slave-1l3l krb5kdc[26291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:54 dist-test-slave-1l3l krb5kdc[26291](info): set up 2 sockets
Aug 12 14:31:54 dist-test-slave-1l3l krb5kdc[26291](info): commencing operation
krb5kdc: starting...
W20260812 14:31:55.521317 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:55 dist-test-slave-1l3l krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:31:55.646060 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:55.729598 26310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:55.729833 26310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:55.729863 26310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:55.731050 26310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:55.731101 26310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:55.731120 26310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:55.731137 26310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:55.731151 26310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:55.732699 26310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:55.732963 26310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:55.733183 26310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:55.738104 26325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:55.738168 26310 server_base.cc:1034] running on GCE node
W20260812 14:31:55.738087 26322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:55.738087 26321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:55.738581 26310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:55.738883 26310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:55.740025 26310 hybrid_clock.cc:648] HybridClock initialized: now 1786545115740012 us; error 26 us; skew 500 ppm
Aug 12 14:31:55 dist-test-slave-1l3l krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:55.747903 26310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:31:55.748348 26310 webserver.cc:460] Webserver started at http://127.0.0.1:41813/ using document root <none> and password file <none>
I20260812 14:31:55.748556 26310 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:55.748606 26310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:55.748734 26310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:55.749831 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "031788037fee45159d9920b85a4bddbc"
format_stamp: "Formatted at 2026-08-12 14:31:55 on dist-test-slave-1l3l"
server_key: "a50f863e23ac652ecd6e0d928c466179"
server_key_iv: "9301cc1ead235e8dc28929e1f82f74d4"
server_key_version: "encryptionkey@0"
I20260812 14:31:55.750178 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "031788037fee45159d9920b85a4bddbc"
format_stamp: "Formatted at 2026-08-12 14:31:55 on dist-test-slave-1l3l"
server_key: "a50f863e23ac652ecd6e0d928c466179"
server_key_iv: "9301cc1ead235e8dc28929e1f82f74d4"
server_key_version: "encryptionkey@0"
I20260812 14:31:55.751448 26310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:31:55.752233 26332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:55.752424 26310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:55.752533 26310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "031788037fee45159d9920b85a4bddbc"
format_stamp: "Formatted at 2026-08-12 14:31:55 on dist-test-slave-1l3l"
server_key: "a50f863e23ac652ecd6e0d928c466179"
server_key_iv: "9301cc1ead235e8dc28929e1f82f74d4"
server_key_version: "encryptionkey@0"
I20260812 14:31:55.752619 26310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:55.762625 26310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:55.764719 26310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:55.764829 26310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:55.768712 26310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38551
I20260812 14:31:55.768751 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38551 every 8 connection(s)
I20260812 14:31:55.769109 26310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:55.769974 26427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:55.771742 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26310
I20260812 14:31:55.771860 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:55.772043 24247 external_mini_cluster.cc:1428] Setting key 8f25ac1409864f04e74427b8a66c4b53
I20260812 14:31:55.772595 26427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc: Bootstrap starting.
I20260812 14:31:55.773192 26427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:55.773460 26427 log.cc:824] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:55.774112 26427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc: No bootstrap required, opened a new log
I20260812 14:31:55.775915 26427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031788037fee45159d9920b85a4bddbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260812 14:31:55.776058 26427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:55.776093 26427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031788037fee45159d9920b85a4bddbc, State: Initialized, Role: FOLLOWER
I20260812 14:31:55.776218 26427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031788037fee45159d9920b85a4bddbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260812 14:31:55.776290 26427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:55.776324 26427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:55.776369 26427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:55.776935 26427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031788037fee45159d9920b85a4bddbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260812 14:31:55.777042 26427 leader_election.cc:302] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 031788037fee45159d9920b85a4bddbc; no voters:
I20260812 14:31:55.777199 26427 leader_election.cc:288] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:55.777237 26434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:55.777359 26434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [term 1 LEADER]: Becoming Leader. State: Replica: 031788037fee45159d9920b85a4bddbc, State: Running, Role: LEADER
I20260812 14:31:55.777447 26434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031788037fee45159d9920b85a4bddbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260812 14:31:55.777537 26427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:55.777817 26435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 031788037fee45159d9920b85a4bddbc. Latest consensus state: current_term: 1 leader_uuid: "031788037fee45159d9920b85a4bddbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031788037fee45159d9920b85a4bddbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } } }
I20260812 14:31:55.777907 26435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:55.778098 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "031788037fee45159d9920b85a4bddbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031788037fee45159d9920b85a4bddbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } } }
I20260812 14:31:55.778154 26442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:55.778170 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc [sys.catalog]: This master's current role is: LEADER
Aug 12 14:31:55 dist-test-slave-1l3l krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:31:55.778817 26442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:55.782440 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:55.772624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58369 (local address 127.0.0.1:38551)
0812 14:31:55.772801 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:55.772809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:55.772834 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:31:55.773317 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:55.773327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:55.773379 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:55.773509 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:55.774174 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:55.775114 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:55.775913 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:55.776430 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:55.776532 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:55.779281 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:55.779286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:55.779286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:55.779298 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:55.780987 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:55.781355 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:55.781359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:55.781359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:55.781448 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:55.781653 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:55.781654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:55.781654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:55.781888 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:55.782049 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:55.782167 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 14:31:55.782263 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260812 14:31:55.783403 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:31:55.784776 26442 catalog_manager.cc:1347] Generated new cluster ID: 355ae02e089540799fa9ba9f1ff906ec
I20260812 14:31:55.784828 26442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:55.798816 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:55.785098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36751 (local address 127.0.0.1:38551)
0812 14:31:55.785140 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:55.785143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:55.791881 (+ 6738us) server_negotiation.cc:407] Connection header received
0812 14:31:55.791940 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:55.791942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:55.791953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:55.792004 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:55.792585 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:55.793349 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:55.794282 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:55.794638 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:55.794701 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:55.796065 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:55.796066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:55.796066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:31:55.796075 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:55.797640 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:55.798040 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:55.798041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:55.798042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:55.798103 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:31:55.798295 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:31:55.798295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:31:55.798296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:31:55.798436 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:55.798571 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:55.798653 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 14:31:55.798718 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:31:55.803737 26442 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:55.804348 26442 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:55.807439 26442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 031788037fee45159d9920b85a4bddbc: Generated new TSK 0
I20260812 14:31:55.807632 26442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:31:55.809842 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26310
W20260812 14:31:55.828431 26284 connection.cc:535] client connection to 127.0.0.1:38551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38551): Connection reset by peer (error 104)
2026-08-12T14:31:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:55 dist-test-slave-1l3l krb5kdc[26478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:55 dist-test-slave-1l3l krb5kdc[26478](info): set up 2 sockets
Aug 12 14:31:55 dist-test-slave-1l3l krb5kdc[26478](info): commencing operation
krb5kdc: starting...
W20260812 14:31:57.000735 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:57 dist-test-slave-1l3l krb5kdc[26478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:31:57.118484 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:57.201792 26500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:57.202006 26500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:57.202037 26500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:57.203195 26500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:57.203240 26500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:57.203258 26500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:57.203274 26500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:57.203290 26500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:57.204811 26500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:57.205065 26500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:57.205276 26500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:57.209894 26513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:57.209908 26511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:57.209908 26515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:57.210170 26500 server_base.cc:1034] running on GCE node
I20260812 14:31:57.210436 26500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:57.210680 26500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:57.211767 26500 hybrid_clock.cc:648] HybridClock initialized: now 1786545117211754 us; error 30 us; skew 500 ppm
Aug 12 14:31:57 dist-test-slave-1l3l krb5kdc[26478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:57.214178 26500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:31:57.214656 26500 webserver.cc:460] Webserver started at http://127.0.0.1:45853/ using document root <none> and password file <none>
I20260812 14:31:57.214869 26500 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:57.214918 26500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:57.215042 26500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:57.216123 26500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4"
format_stamp: "Formatted at 2026-08-12 14:31:57 on dist-test-slave-1l3l"
server_key: "26fd9e11d74d32e0b2fa9894d2966e1f"
server_key_iv: "3a3a9840ea5eb4f41f9acff13963f899"
server_key_version: "encryptionkey@0"
I20260812 14:31:57.216465 26500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4"
format_stamp: "Formatted at 2026-08-12 14:31:57 on dist-test-slave-1l3l"
server_key: "26fd9e11d74d32e0b2fa9894d2966e1f"
server_key_iv: "3a3a9840ea5eb4f41f9acff13963f899"
server_key_version: "encryptionkey@0"
I20260812 14:31:57.217696 26500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:31:57.218344 26526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:57.218508 26500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:31:57.218590 26500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4"
format_stamp: "Formatted at 2026-08-12 14:31:57 on dist-test-slave-1l3l"
server_key: "26fd9e11d74d32e0b2fa9894d2966e1f"
server_key_iv: "3a3a9840ea5eb4f41f9acff13963f899"
server_key_version: "encryptionkey@0"
I20260812 14:31:57.218650 26500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:57.231580 26500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:57.233283 26500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:57.233422 26500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:57.237246 26500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35997
I20260812 14:31:57.237280 26608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35997 every 8 connection(s)
I20260812 14:31:57.237598 26500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:57.238407 26611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:57.240604 26611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4: Bootstrap starting.
I20260812 14:31:57.241163 26611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:57.241421 26611 log.cc:824] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:57.242061 26611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4: No bootstrap required, opened a new log
I20260812 14:31:57.243557 26611 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 14:31:57.243736 26611 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:57.243775 26611 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c4418642d144c1a38e2bb4ee7bb7d4, State: Initialized, Role: FOLLOWER
I20260812 14:31:57.243881 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26500
I20260812 14:31:57.243919 26611 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 14:31:57.243988 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:57.244001 26611 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:57.244031 26611 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:57.244084 26611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:57.244166 24247 external_mini_cluster.cc:1428] Setting key 0cd7b43bfd6718ca98d0b2bef8bc4435
I20260812 14:31:57.244750 26611 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 14:31:57.244855 26611 leader_election.cc:302] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9c4418642d144c1a38e2bb4ee7bb7d4; no voters:
I20260812 14:31:57.245012 26611 leader_election.cc:288] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:57.245056 26619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:57.245211 26619 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [term 1 LEADER]: Becoming Leader. State: Replica: c9c4418642d144c1a38e2bb4ee7bb7d4, State: Running, Role: LEADER
I20260812 14:31:57.245265 26611 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:57.245301 26619 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 14:31:57.245662 26619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c4418642d144c1a38e2bb4ee7bb7d4. Latest consensus state: current_term: 1 leader_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } } }
I20260812 14:31:57.245738 26619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:57.245718 26620 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4418642d144c1a38e2bb4ee7bb7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } } }
I20260812 14:31:57.245966 26620 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:57.246021 26632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:57.246547 26632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:57.248320 26632 catalog_manager.cc:1347] Generated new cluster ID: 35f49ce567c141028aee360c69f9b139
I20260812 14:31:57.248377 26632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:57.250344 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:57.244607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39283 (local address 127.0.0.1:35997)
0812 14:31:57.244817 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:57.244827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:57.244867 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:31:57.244985 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:57.244992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:57.245049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:57.245451 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:57.245541 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:57.247291 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:57.248287 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:57.248890 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:57.249005 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:57.249683 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:57.249687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:57.249687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:57.249894 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:57.249932 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:57.249947 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:57.250035 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:57.250114 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:31:57.250143 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260812 14:31:57.251344 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:31:57.267093 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:57.252714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57001 (local address 127.0.0.1:35997)
0812 14:31:57.252745 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:57.252746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:57.263860 (+ 11114us) server_negotiation.cc:407] Connection header received
0812 14:31:57.263922 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:57.263922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:57.263932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:57.263978 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:57.264147 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:57.264861 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:57.265855 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:31:57.266163 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:31:57.266212 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:31:57.266867 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:57.266869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:57.266869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:57.266873 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:57.266888 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:57.266892 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:57.266943 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:57.267010 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 14:31:57.267022 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:31:57.269547 26632 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:57.270025 26632 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:57.275846 26632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9c4418642d144c1a38e2bb4ee7bb7d4: Generated new TSK 0
I20260812 14:31:57.276011 26632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:31:57.278146 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26500
W20260812 14:31:57.292770 26476 connection.cc:535] client connection to 127.0.0.1:35997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35997): Connection reset by peer (error 104)
2026-08-12T14:31:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:57 dist-test-slave-1l3l krb5kdc[26670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:57 dist-test-slave-1l3l krb5kdc[26670](info): set up 2 sockets
Aug 12 14:31:57 dist-test-slave-1l3l krb5kdc[26670](info): commencing operation
krb5kdc: starting...
W20260812 14:31:58.445139 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:58 dist-test-slave-1l3l krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:31:58.564306 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:31:58.647202 26690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:31:58.647444 26690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:31:58.647469 26690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:31:58.648653 26690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:31:58.648693 26690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:31:58.648706 26690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:31:58.648718 26690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:31:58.648729 26690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:31:58.650187 26690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:31:58.650404 26690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:31:58.650599 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:31:58.655567 26700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:58.655741 26690 server_base.cc:1034] running on GCE node
W20260812 14:31:58.655647 26698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:31:58.655567 26697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:31:58.656124 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:31:58.656440 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:31:58.657608 26690 hybrid_clock.cc:648] HybridClock initialized: now 1786545118657590 us; error 30 us; skew 500 ppm
Aug 12 14:31:58 dist-test-slave-1l3l krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:31:58.660275 26690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:31:58.660771 26690 webserver.cc:460] Webserver started at http://127.0.0.1:34219/ using document root <none> and password file <none>
I20260812 14:31:58.661015 26690 fs_manager.cc:359] Metadata directory not provided
I20260812 14:31:58.661069 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:31:58.661207 26690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:31:58.662298 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "bd1eedfae92f4e8697dd40d1f5108f7c"
format_stamp: "Formatted at 2026-08-12 14:31:58 on dist-test-slave-1l3l"
server_key: "1186f851670a21c987cf681df154c096"
server_key_iv: "203f08fa9bfda29cf643ce78d23361fd"
server_key_version: "encryptionkey@0"
I20260812 14:31:58.662662 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "bd1eedfae92f4e8697dd40d1f5108f7c"
format_stamp: "Formatted at 2026-08-12 14:31:58 on dist-test-slave-1l3l"
server_key: "1186f851670a21c987cf681df154c096"
server_key_iv: "203f08fa9bfda29cf643ce78d23361fd"
server_key_version: "encryptionkey@0"
I20260812 14:31:58.664022 26690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:31:58.664760 26715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:31:58.664917 26690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:31:58.664995 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "bd1eedfae92f4e8697dd40d1f5108f7c"
format_stamp: "Formatted at 2026-08-12 14:31:58 on dist-test-slave-1l3l"
server_key: "1186f851670a21c987cf681df154c096"
server_key_iv: "203f08fa9bfda29cf643ce78d23361fd"
server_key_version: "encryptionkey@0"
I20260812 14:31:58.665061 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:31:58.668503 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:31:58.670702 26690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:31:58.670840 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:31:58.674971 26690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34733
I20260812 14:31:58.675012 26805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34733 every 8 connection(s)
I20260812 14:31:58.675434 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:31:58.676463 26806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:31:58.678769 26806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c: Bootstrap starting.
I20260812 14:31:58.679244 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26690
I20260812 14:31:58.679363 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:31:58.679417 26806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c: Neither blocks nor log segments found. Creating new log.
I20260812 14:31:58.679575 24247 external_mini_cluster.cc:1428] Setting key 3bacd27b4d200be3ade54237db7eeabc
I20260812 14:31:58.679750 26806 log.cc:824] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:31:58.680550 26806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c: No bootstrap required, opened a new log
I20260812 14:31:58.682010 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:58.680079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49055 (local address 127.0.0.1:34733)
0812 14:31:58.680256 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:58.680267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:58.680307 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:31:58.680784 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:58.680792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:58.680820 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:58.680940 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:58.681594 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:58.681598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:58.681598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:58.681602 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:58.681644 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:58.681654 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:58.681722 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:58.681783 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:31:58.681816 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260812 14:31:58.682930 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:31:58.682826 26806 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260812 14:31:58.682984 26806 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:31:58.683020 26806 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1eedfae92f4e8697dd40d1f5108f7c, State: Initialized, Role: FOLLOWER
I20260812 14:31:58.683137 26806 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260812 14:31:58.683218 26806 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:31:58.683248 26806 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:31:58.683296 26806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:31:58.683926 26806 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260812 14:31:58.684037 26806 leader_election.cc:302] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1eedfae92f4e8697dd40d1f5108f7c; no voters:
I20260812 14:31:58.684224 26806 leader_election.cc:288] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:31:58.684269 26827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:31:58.684419 26827 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [term 1 LEADER]: Becoming Leader. State: Replica: bd1eedfae92f4e8697dd40d1f5108f7c, State: Running, Role: LEADER
I20260812 14:31:58.684489 26806 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:31:58.684551 26827 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260812 14:31:58.685070 26828 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } } }
I20260812 14:31:58.685179 26828 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:58.685070 26829 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1eedfae92f4e8697dd40d1f5108f7c. Latest consensus state: current_term: 1 leader_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eedfae92f4e8697dd40d1f5108f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } } }
I20260812 14:31:58.685395 26829 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c [sys.catalog]: This master's current role is: LEADER
I20260812 14:31:58.685467 26840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:31:58.686080 26840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:31:58.686295 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:31:58.684575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51463 (local address 127.0.0.1:34733)
0812 14:31:58.684613 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:31:58.684616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:31:58.684735 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:31:58.684800 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:31:58.684801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:31:58.684806 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:31:58.684863 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:31:58.686058 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:31:58.686060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:31:58.686060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:31:58.686064 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:31:58.686081 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:31:58.686085 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:31:58.686115 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 14:31:58.686227 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 14:31:58.686241 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:31:58.687700 26840 catalog_manager.cc:1347] Generated new cluster ID: b5c44b8e1f3c49a2b13182c832034d0b
I20260812 14:31:58.687773 26840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:31:58.709066 26840 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:31:58.709286 26840 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:31:58.712191 26840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd1eedfae92f4e8697dd40d1f5108f7c: Generated new TSK 0
I20260812 14:31:58.712424 26840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:31:58.714732 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26690
2026-08-12T14:31:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:31:58 dist-test-slave-1l3l krb5kdc[26867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:31:58 dist-test-slave-1l3l krb5kdc[26867](info): set up 2 sockets
Aug 12 14:31:58 dist-test-slave-1l3l krb5kdc[26867](info): commencing operation
krb5kdc: starting...
W20260812 14:31:59.896353 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:31:59 dist-test-slave-1l3l krb5kdc[26867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:00.019448 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:00.102077 26891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:00.102278 26891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:00.102300 26891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:00.103444 26891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:00.103487 26891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:00.103500 26891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:00.103511 26891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:00.103523 26891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:00.105020 26891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:00.105237 26891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:00.105464 26891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:00.110172 26901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:00.110195 26904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:00.110270 26891 server_base.cc:1034] running on GCE node
W20260812 14:32:00.110216 26900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:00.110637 26891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:00.110945 26891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:00.112099 26891 hybrid_clock.cc:648] HybridClock initialized: now 1786545120112081 us; error 34 us; skew 500 ppm
Aug 12 14:32:00 dist-test-slave-1l3l krb5kdc[26867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:00.114593 26891 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:00.115025 26891 webserver.cc:460] Webserver started at http://127.0.0.1:40645/ using document root <none> and password file <none>
I20260812 14:32:00.115252 26891 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:00.115301 26891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:00.115437 26891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:00.116506 26891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "5ae6a24f098d420fa566dbff86dcba31"
format_stamp: "Formatted at 2026-08-12 14:32:00 on dist-test-slave-1l3l"
server_key: "2364c52dc85dd93e5d2425d19ccd5137"
server_key_iv: "d95e3227b08466afd0600f1d4ea988b9"
server_key_version: "encryptionkey@0"
I20260812 14:32:00.116837 26891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "5ae6a24f098d420fa566dbff86dcba31"
format_stamp: "Formatted at 2026-08-12 14:32:00 on dist-test-slave-1l3l"
server_key: "2364c52dc85dd93e5d2425d19ccd5137"
server_key_iv: "d95e3227b08466afd0600f1d4ea988b9"
server_key_version: "encryptionkey@0"
I20260812 14:32:00.118120 26891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:32:00.118765 26918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:00.118904 26891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:32:00.118988 26891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "5ae6a24f098d420fa566dbff86dcba31"
format_stamp: "Formatted at 2026-08-12 14:32:00 on dist-test-slave-1l3l"
server_key: "2364c52dc85dd93e5d2425d19ccd5137"
server_key_iv: "d95e3227b08466afd0600f1d4ea988b9"
server_key_version: "encryptionkey@0"
I20260812 14:32:00.119065 26891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:00.129964 26891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:00.132125 26891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:00.132264 26891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:00.136193 26891 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41003
I20260812 14:32:00.136229 26997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41003 every 8 connection(s)
I20260812 14:32:00.136576 26891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:00.137416 26998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:00.139830 26998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31: Bootstrap starting.
I20260812 14:32:00.140417 26998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:00.140691 26998 log.cc:824] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:00.141428 26998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31: No bootstrap required, opened a new log
I20260812 14:32:00.143049 26998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae6a24f098d420fa566dbff86dcba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260812 14:32:00.143194 26998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:00.143215 26998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae6a24f098d420fa566dbff86dcba31, State: Initialized, Role: FOLLOWER
I20260812 14:32:00.143347 26998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae6a24f098d420fa566dbff86dcba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260812 14:32:00.143415 26998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:00.143445 26998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:00.143504 26998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:00.144078 26998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae6a24f098d420fa566dbff86dcba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260812 14:32:00.144176 26998 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae6a24f098d420fa566dbff86dcba31; no voters:
I20260812 14:32:00.144330 26998 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:00.144378 27004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:00.144570 27004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [term 1 LEADER]: Becoming Leader. State: Replica: 5ae6a24f098d420fa566dbff86dcba31, State: Running, Role: LEADER
I20260812 14:32:00.144594 26998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:00.144752 27004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae6a24f098d420fa566dbff86dcba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260812 14:32:00.145005 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 26891
I20260812 14:32:00.145141 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:00.145376 24247 external_mini_cluster.cc:1428] Setting key 094eef07e277f314770e0ffbb6e77b1d
I20260812 14:32:00.146196 27005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae6a24f098d420fa566dbff86dcba31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae6a24f098d420fa566dbff86dcba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } } }
I20260812 14:32:00.146301 27005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:00.146487 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae6a24f098d420fa566dbff86dcba31. Latest consensus state: current_term: 1 leader_uuid: "5ae6a24f098d420fa566dbff86dcba31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae6a24f098d420fa566dbff86dcba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } } }
I20260812 14:32:00.146494 27030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:00.146607 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:00.147161 27030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:00.149121 27030 catalog_manager.cc:1347] Generated new cluster ID: f2f18aa2808044b19fb376c909ce0373
I20260812 14:32:00.149180 27030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:32:00 dist-test-slave-1l3l krb5kdc[26867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:32:00.156490 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:00.145844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58721 (local address 127.0.0.1:41003)
0812 14:32:00.146033 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:00.146043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:00.146883 (+ 840us) server_negotiation.cc:407] Connection header received
0812 14:32:00.147065 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:00.147083 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:00.147144 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:00.147296 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:00.148129 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:00.149293 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:00.150035 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:00.150486 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:00.150585 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:00.152893 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:00.152897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:00.152897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:00.152910 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:00.154952 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:00.155280 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:00.155283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:00.155283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:00.155415 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:00.155603 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:00.155603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:00.155604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:00.155876 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:00.156057 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:00.156127 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:32:00.156265 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260812 14:32:00.157313 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:00.167136 27030 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:00.167881 27030 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:00.169925 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:00.159991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53411 (local address 127.0.0.1:41003)
0812 14:32:00.160027 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:00.160029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:00.163900 (+ 3871us) server_negotiation.cc:407] Connection header received
0812 14:32:00.163970 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:00.163971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:00.163984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:00.164034 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:00.164579 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:00.165096 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:00.165908 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:00.166143 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:00.166202 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:00.167555 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:00.167556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:00.167557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:00.167564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:00.168903 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:00.169225 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:00.169226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:00.169227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:00.169281 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:00.169450 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:00.169451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:00.169451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:00.169557 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:00.169652 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:00.169750 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:32:00.169838 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:32:00.173316 27030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae6a24f098d420fa566dbff86dcba31: Generated new TSK 0
I20260812 14:32:00.173524 27030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:00.176524 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 26891
2026-08-12T14:32:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:00 dist-test-slave-1l3l krb5kdc[27047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:00 dist-test-slave-1l3l krb5kdc[27047](info): set up 2 sockets
Aug 12 14:32:00 dist-test-slave-1l3l krb5kdc[27047](info): commencing operation
krb5kdc: starting...
W20260812 14:32:01.358835 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:01 dist-test-slave-1l3l krb5kdc[27047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:01.476336 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:01.558786 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:01.559019 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:01.559043 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:01.560226 27072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:01.560264 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:01.560276 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:01.560288 27072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:01.560298 27072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:01.561757 27072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:01.561964 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:01.562152 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:01.567022 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:01.567010 27080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:01.567180 27072 server_base.cc:1034] running on GCE node
W20260812 14:32:01.567137 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:01.567468 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:01.567807 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:01.568950 27072 hybrid_clock.cc:648] HybridClock initialized: now 1786545121568938 us; error 31 us; skew 500 ppm
Aug 12 14:32:01 dist-test-slave-1l3l krb5kdc[27047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:01.575790 27072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:01.576287 27072 webserver.cc:460] Webserver started at http://127.0.0.1:38197/ using document root <none> and password file <none>
I20260812 14:32:01.576546 27072 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:01.576601 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:01.576714 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:01.577826 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "8669f29e45c54a0cab3e7d7b877511e5"
format_stamp: "Formatted at 2026-08-12 14:32:01 on dist-test-slave-1l3l"
server_key: "27fc3105c46c9eaba4e344f14611d120"
server_key_iv: "6849ac7bfbe14bca9bc7299527ca13ec"
server_key_version: "encryptionkey@0"
I20260812 14:32:01.578177 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "8669f29e45c54a0cab3e7d7b877511e5"
format_stamp: "Formatted at 2026-08-12 14:32:01 on dist-test-slave-1l3l"
server_key: "27fc3105c46c9eaba4e344f14611d120"
server_key_iv: "6849ac7bfbe14bca9bc7299527ca13ec"
server_key_version: "encryptionkey@0"
I20260812 14:32:01.579649 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:32:01.580500 27092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:01.580672 27072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:32:01.580750 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "8669f29e45c54a0cab3e7d7b877511e5"
format_stamp: "Formatted at 2026-08-12 14:32:01 on dist-test-slave-1l3l"
server_key: "27fc3105c46c9eaba4e344f14611d120"
server_key_iv: "6849ac7bfbe14bca9bc7299527ca13ec"
server_key_version: "encryptionkey@0"
I20260812 14:32:01.580822 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:01.607964 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:01.610270 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:01.610474 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:01.614684 27072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38693
I20260812 14:32:01.614794 27177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38693 every 8 connection(s)
I20260812 14:32:01.615096 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:01.616061 27178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:01.618427 27178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5: Bootstrap starting.
I20260812 14:32:01.618974 27178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:01.619216 27178 log.cc:824] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:01.619989 27178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5: No bootstrap required, opened a new log
I20260812 14:32:01.621627 27178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8669f29e45c54a0cab3e7d7b877511e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260812 14:32:01.621827 27178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:01.621850 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27072
I20260812 14:32:01.621866 27178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8669f29e45c54a0cab3e7d7b877511e5, State: Initialized, Role: FOLLOWER
I20260812 14:32:01.621976 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:01.622026 27178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8669f29e45c54a0cab3e7d7b877511e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260812 14:32:01.622110 27178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:01.622148 27178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:01.622169 24247 external_mini_cluster.cc:1428] Setting key 0dd61b2fee46b4818ec96edb6c3bfb0a
I20260812 14:32:01.622196 27178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:01.622898 27178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8669f29e45c54a0cab3e7d7b877511e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260812 14:32:01.623003 27178 leader_election.cc:302] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8669f29e45c54a0cab3e7d7b877511e5; no voters:
I20260812 14:32:01.623196 27178 leader_election.cc:288] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:01.623245 27188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:01.623407 27188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8669f29e45c54a0cab3e7d7b877511e5, State: Running, Role: LEADER
I20260812 14:32:01.623492 27178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:01.623548 27188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8669f29e45c54a0cab3e7d7b877511e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260812 14:32:01.623898 27188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8669f29e45c54a0cab3e7d7b877511e5. Latest consensus state: current_term: 1 leader_uuid: "8669f29e45c54a0cab3e7d7b877511e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8669f29e45c54a0cab3e7d7b877511e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } } }
I20260812 14:32:01.623980 27188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:01.623910 27194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8669f29e45c54a0cab3e7d7b877511e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8669f29e45c54a0cab3e7d7b877511e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } } }
I20260812 14:32:01.624155 27194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:01.624320 27198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:01.624944 27198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:01.626922 27198 catalog_manager.cc:1347] Generated new cluster ID: acfb4de4e86248e0a357e9d14874a7f7
I20260812 14:32:01.626986 27198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:01.628954 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:01.622966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52469 (local address 127.0.0.1:38693)
0812 14:32:01.623170 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:01.623180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:01.623208 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:32:01.623330 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:01.623337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:01.623392 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:01.623848 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:01.623966 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:01.625767 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:01.626855 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:01.627312 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:01.627415 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:01.628160 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:01.628164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:01.628164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:01.628399 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:01.628432 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:01.628447 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:01.628521 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:01.628595 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 14:32:01.628754 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260812 14:32:01.629913 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:01.634210 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:01.631243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60155 (local address 127.0.0.1:38693)
0812 14:32:01.631279 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:01.631281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:01.631398 (+ 117us) server_negotiation.cc:407] Connection header received
0812 14:32:01.631450 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:01.631451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:01.631461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:01.631506 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:01.631626 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:01.632399 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:01.633101 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:01.633306 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:01.633358 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:01.633988 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:01.633989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:01.633990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:01.633995 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:01.634010 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:01.634015 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:01.634060 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:01.634111 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:32:01.634136 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:32:01.651085 27198 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:01.651937 27198 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:01.676843 27198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8669f29e45c54a0cab3e7d7b877511e5: Generated new TSK 0
I20260812 14:32:01.677102 27198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:01.682255 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27072
2026-08-12T14:32:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:01 dist-test-slave-1l3l krb5kdc[27235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:01 dist-test-slave-1l3l krb5kdc[27235](info): set up 2 sockets
Aug 12 14:32:01 dist-test-slave-1l3l krb5kdc[27235](info): commencing operation
krb5kdc: starting...
W20260812 14:32:02.868657 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:02 dist-test-slave-1l3l krb5kdc[27235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:02.994207 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:03.079370 27254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:03.079571 27254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:03.079594 27254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:03.080773 27254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:03.080819 27254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:03.080832 27254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:03.080847 27254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:03.080880 27254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:03.082402 27254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:03.082641 27254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:03.082856 27254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:03.087533 27262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:03.087541 27259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:03.087541 27264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:03.087905 27254 server_base.cc:1034] running on GCE node
I20260812 14:32:03.088115 27254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:03.088387 27254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:03.089535 27254 hybrid_clock.cc:648] HybridClock initialized: now 1786545123089517 us; error 30 us; skew 500 ppm
Aug 12 14:32:03 dist-test-slave-1l3l krb5kdc[27235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:03.091993 27254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:03.092442 27254 webserver.cc:460] Webserver started at http://127.0.0.1:33481/ using document root <none> and password file <none>
I20260812 14:32:03.092648 27254 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:03.092694 27254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:03.092820 27254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:03.093868 27254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5"
format_stamp: "Formatted at 2026-08-12 14:32:03 on dist-test-slave-1l3l"
server_key: "b59bbb1bc706e0bc9c4a8e3a86a3d87e"
server_key_iv: "a1d1c5127ef1a7b054c57a9262c3a185"
server_key_version: "encryptionkey@0"
I20260812 14:32:03.094216 27254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5"
format_stamp: "Formatted at 2026-08-12 14:32:03 on dist-test-slave-1l3l"
server_key: "b59bbb1bc706e0bc9c4a8e3a86a3d87e"
server_key_iv: "a1d1c5127ef1a7b054c57a9262c3a185"
server_key_version: "encryptionkey@0"
I20260812 14:32:03.095570 27254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:32:03.096449 27272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:03.096659 27254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:03.096751 27254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5"
format_stamp: "Formatted at 2026-08-12 14:32:03 on dist-test-slave-1l3l"
server_key: "b59bbb1bc706e0bc9c4a8e3a86a3d87e"
server_key_iv: "a1d1c5127ef1a7b054c57a9262c3a185"
server_key_version: "encryptionkey@0"
I20260812 14:32:03.096817 27254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:03.100131 27254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:03.102205 27254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:03.102334 27254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:03.106153 27254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44797
I20260812 14:32:03.106186 27361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44797 every 8 connection(s)
I20260812 14:32:03.106475 27254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:03.107415 27362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:03.109640 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27254
I20260812 14:32:03.109647 27362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5: Bootstrap starting.
I20260812 14:32:03.109748 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:03.109920 24247 external_mini_cluster.cc:1428] Setting key 9fb19131ed2cca96b660a410ac89f254
I20260812 14:32:03.110276 27362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:03.110687 27362 log.cc:824] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:03.111439 27362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5: No bootstrap required, opened a new log
I20260812 14:32:03.112432 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:03.110389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42895 (local address 127.0.0.1:44797)
0812 14:32:03.110629 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:03.110639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:03.110676 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:32:03.111204 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:03.111213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:03.111248 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:03.111402 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:03.112018 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:03.112022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:03.112022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:03.112027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:03.112072 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:03.112087 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:03.112178 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:03.112208 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 14:32:03.112247 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260812 14:32:03.113446 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:03.113807 27362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44797 } }
I20260812 14:32:03.113986 27362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:03.114027 27362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99122ad5eb3c43728ea7cd52c2e7f4e5, State: Initialized, Role: FOLLOWER
I20260812 14:32:03.114144 27362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44797 } }
I20260812 14:32:03.114249 27362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:03.114287 27362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:03.114334 27362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:03.115041 27362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44797 } }
I20260812 14:32:03.115149 27362 leader_election.cc:302] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99122ad5eb3c43728ea7cd52c2e7f4e5; no voters:
I20260812 14:32:03.115307 27362 leader_election.cc:288] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:03.115527 27381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:03.115590 27362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:03.115744 27381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [term 1 LEADER]: Becoming Leader. State: Replica: 99122ad5eb3c43728ea7cd52c2e7f4e5, State: Running, Role: LEADER
I20260812 14:32:03.115856 27381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44797 } }
I20260812 14:32:03.116211 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:03.115019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47449 (local address 127.0.0.1:44797)
0812 14:32:03.115062 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:03.115064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:03.115167 (+ 103us) server_negotiation.cc:407] Connection header received
0812 14:32:03.115236 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:03.115237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:03.115243 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:03.115308 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:03.115935 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:03.115936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:03.115936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:03.115940 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:03.115955 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:03.115959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:03.116005 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:03.116144 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:32:03.116157 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:32:03.116283 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99122ad5eb3c43728ea7cd52c2e7f4e5. Latest consensus state: current_term: 1 leader_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44797 } } }
I20260812 14:32:03.116355 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:03.116601 27395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:03.116852 27379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99122ad5eb3c43728ea7cd52c2e7f4e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44797 } } }
I20260812 14:32:03.117136 27379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:03.117372 27395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:03.118836 27395 catalog_manager.cc:1347] Generated new cluster ID: 03cb8ff26d374d009c4fbfc63b781327
I20260812 14:32:03.118882 27395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:03.134044 27395 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:03.134253 27395 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:03.142789 27395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99122ad5eb3c43728ea7cd52c2e7f4e5: Generated new TSK 0
I20260812 14:32:03.142983 27395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:03.152617 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27254
2026-08-12T14:32:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:03 dist-test-slave-1l3l krb5kdc[27414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:03 dist-test-slave-1l3l krb5kdc[27414](info): set up 2 sockets
Aug 12 14:32:03 dist-test-slave-1l3l krb5kdc[27414](info): commencing operation
krb5kdc: starting...
W20260812 14:32:04.341897 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:04 dist-test-slave-1l3l krb5kdc[27414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:04.456437 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:04.537560 27432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:04.537755 27432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:04.537777 27432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:04.538931 27432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:04.538975 27432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:04.538988 27432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:04.539000 27432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:04.539013 27432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:04.540534 27432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:04.540750 27432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:04.540977 27432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:04.545615 27447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:04.545742 27432 server_base.cc:1034] running on GCE node
W20260812 14:32:04.545624 27444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:04.545841 27445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:04.546134 27432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:04.546418 27432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:04.547565 27432 hybrid_clock.cc:648] HybridClock initialized: now 1786545124547552 us; error 31 us; skew 500 ppm
Aug 12 14:32:04 dist-test-slave-1l3l krb5kdc[27414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:04.549948 27432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:04.550383 27432 webserver.cc:460] Webserver started at http://127.0.0.1:46031/ using document root <none> and password file <none>
I20260812 14:32:04.550638 27432 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:04.550688 27432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:04.550815 27432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:04.551883 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "f3228c11bc61440bb80f1f484e0a1def"
format_stamp: "Formatted at 2026-08-12 14:32:04 on dist-test-slave-1l3l"
server_key: "c3c6b5ae6446be0daf7b55046c6a34f9"
server_key_iv: "72aad2ade82f24b229ab92b00bf5a81b"
server_key_version: "encryptionkey@0"
I20260812 14:32:04.552222 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "f3228c11bc61440bb80f1f484e0a1def"
format_stamp: "Formatted at 2026-08-12 14:32:04 on dist-test-slave-1l3l"
server_key: "c3c6b5ae6446be0daf7b55046c6a34f9"
server_key_iv: "72aad2ade82f24b229ab92b00bf5a81b"
server_key_version: "encryptionkey@0"
I20260812 14:32:04.553476 27432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:04.554258 27464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:04.554490 27432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:04.554586 27432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "f3228c11bc61440bb80f1f484e0a1def"
format_stamp: "Formatted at 2026-08-12 14:32:04 on dist-test-slave-1l3l"
server_key: "c3c6b5ae6446be0daf7b55046c6a34f9"
server_key_iv: "72aad2ade82f24b229ab92b00bf5a81b"
server_key_version: "encryptionkey@0"
I20260812 14:32:04.554652 27432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:04.570801 27432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:04.572692 27432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:04.572835 27432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:04.576786 27432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34237
I20260812 14:32:04.576820 27552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34237 every 8 connection(s)
I20260812 14:32:04.577147 27432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:04.577958 27553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:04.580209 27553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def: Bootstrap starting.
I20260812 14:32:04.580720 27553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:04.580964 27553 log.cc:824] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:04.581337 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27432
I20260812 14:32:04.581471 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:04.581674 24247 external_mini_cluster.cc:1428] Setting key e9ec9f844e6c942785517f2e46401ed3
I20260812 14:32:04.581754 27553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def: No bootstrap required, opened a new log
I20260812 14:32:04.583887 27553 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3228c11bc61440bb80f1f484e0a1def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260812 14:32:04.584059 27553 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:04.584098 27553 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3228c11bc61440bb80f1f484e0a1def, State: Initialized, Role: FOLLOWER
I20260812 14:32:04.584220 27553 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3228c11bc61440bb80f1f484e0a1def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260812 14:32:04.584280 27553 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:04.584309 27553 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:04.584355 27553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:04.584897 27553 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3228c11bc61440bb80f1f484e0a1def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260812 14:32:04.585011 27553 leader_election.cc:302] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3228c11bc61440bb80f1f484e0a1def; no voters:
I20260812 14:32:04.585167 27553 leader_election.cc:288] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:04.585206 27563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:04.585356 27563 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [term 1 LEADER]: Becoming Leader. State: Replica: f3228c11bc61440bb80f1f484e0a1def, State: Running, Role: LEADER
I20260812 14:32:04.585436 27553 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:04.585538 27563 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3228c11bc61440bb80f1f484e0a1def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260812 14:32:04.588061 27563 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3228c11bc61440bb80f1f484e0a1def. Latest consensus state: current_term: 1 leader_uuid: "f3228c11bc61440bb80f1f484e0a1def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3228c11bc61440bb80f1f484e0a1def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } } }
I20260812 14:32:04.588166 27563 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:04.589457 27565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3228c11bc61440bb80f1f484e0a1def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3228c11bc61440bb80f1f484e0a1def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } } }
I20260812 14:32:04.589602 27565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def [sys.catalog]: This master's current role is: LEADER
W20260812 14:32:04.589771 27583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:04.589861 27583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 14:32:04 dist-test-slave-1l3l krb5kdc[27414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:32:04.591879 27570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:04.592080 27570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:04.593742 27570 catalog_manager.cc:1347] Generated new cluster ID: 0ae6259b3f194a1dbe748eb2e97cf165
I20260812 14:32:04.593842 27570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:04.594213 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:04.582326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:42859 (local address 127.0.0.1:34237)
0812 14:32:04.582533 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:04.582543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:04.582569 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:32:04.582984 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:04.582990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:04.583039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:04.583148 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:04.583965 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:04.585333 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:04.586340 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:04.586885 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:04.587305 (+ 420us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:04.590650 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:04.590653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:04.590654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:04.590666 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:04.592320 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:04.592689 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:04.592692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:04.592693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:04.592798 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:04.593077 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:04.593077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:04.593078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:04.593387 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:04.593560 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:04.593895 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 14:32:04.594005 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260812 14:32:04.595424 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:04.602618 27570 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:04.603092 27570 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:04.603978 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:04.596932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:41883 (local address 127.0.0.1:34237)
0812 14:32:04.596969 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:04.596971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:04.597041 (+ 70us) server_negotiation.cc:407] Connection header received
0812 14:32:04.597094 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:04.597095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:04.597104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:04.597168 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:04.597716 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:04.598456 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:04.599405 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:04.599756 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:04.599820 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:04.601116 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:04.601117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:04.601118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:04.601125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:04.602652 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:04.603070 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:04.603071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:04.603072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:04.603124 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:04.603344 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:04.603345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:04.603345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:04.603473 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:04.603565 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:04.603824 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 14:32:04.603893 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:32:04.606832 27570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3228c11bc61440bb80f1f484e0a1def: Generated new TSK 0
I20260812 14:32:04.606994 27570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:04.610572 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27432
W20260812 14:32:04.623448 27412 connection.cc:535] client connection to 127.0.0.1:34237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34237): Connection reset by peer (error 104)
2026-08-12T14:32:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:04 dist-test-slave-1l3l krb5kdc[27601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:04 dist-test-slave-1l3l krb5kdc[27601](info): set up 2 sockets
Aug 12 14:32:04 dist-test-slave-1l3l krb5kdc[27601](info): commencing operation
krb5kdc: starting...
W20260812 14:32:05.787521 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:05 dist-test-slave-1l3l krb5kdc[27601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:05.918519 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:06.002415 27624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:06.002630 27624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:06.002662 27624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:06.004087 27624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:06.004142 27624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:06.004161 27624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:06.004179 27624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:06.004194 27624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:06.005790 27624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:06.006037 27624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:06.006259 27624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:06.011485 27632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:06.011526 27634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:06.011472 27631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:06.012287 27624 server_base.cc:1034] running on GCE node
I20260812 14:32:06.012553 27624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:06.012974 27624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:06.014150 27624 hybrid_clock.cc:648] HybridClock initialized: now 1786545126014131 us; error 38 us; skew 500 ppm
Aug 12 14:32:06 dist-test-slave-1l3l krb5kdc[27601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:06.016847 27624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:06.017288 27624 webserver.cc:460] Webserver started at http://127.0.0.1:35391/ using document root <none> and password file <none>
I20260812 14:32:06.017558 27624 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:06.017613 27624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:06.017740 27624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:06.018878 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "320749e4d38f4a4ba6f9824d34218d55"
format_stamp: "Formatted at 2026-08-12 14:32:06 on dist-test-slave-1l3l"
server_key: "757fa9434f7008abadda19d8906bdc69"
server_key_iv: "7f75752e21c8c3e2c009d6f5006ac719"
server_key_version: "encryptionkey@0"
I20260812 14:32:06.019235 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "320749e4d38f4a4ba6f9824d34218d55"
format_stamp: "Formatted at 2026-08-12 14:32:06 on dist-test-slave-1l3l"
server_key: "757fa9434f7008abadda19d8906bdc69"
server_key_iv: "7f75752e21c8c3e2c009d6f5006ac719"
server_key_version: "encryptionkey@0"
I20260812 14:32:06.020666 27624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:32:06.021533 27649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:06.021747 27624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:06.021847 27624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "320749e4d38f4a4ba6f9824d34218d55"
format_stamp: "Formatted at 2026-08-12 14:32:06 on dist-test-slave-1l3l"
server_key: "757fa9434f7008abadda19d8906bdc69"
server_key_iv: "7f75752e21c8c3e2c009d6f5006ac719"
server_key_version: "encryptionkey@0"
I20260812 14:32:06.021912 27624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:06.040179 27624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:06.042734 27624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:06.042907 27624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:06.046949 27624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33631
I20260812 14:32:06.046993 27734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33631 every 8 connection(s)
I20260812 14:32:06.047359 27624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:06.048302 27735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:06.050666 27735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55: Bootstrap starting.
I20260812 14:32:06.051254 27735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:06.051529 27735 log.cc:824] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:06.052307 27735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55: No bootstrap required, opened a new log
I20260812 14:32:06.053918 27735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320749e4d38f4a4ba6f9824d34218d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260812 14:32:06.054068 27735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:06.054091 27735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320749e4d38f4a4ba6f9824d34218d55, State: Initialized, Role: FOLLOWER
I20260812 14:32:06.054239 27735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320749e4d38f4a4ba6f9824d34218d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260812 14:32:06.054317 27735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:06.054353 27735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:06.054401 27735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:06.054688 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27624
I20260812 14:32:06.054805 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:06.054997 24247 external_mini_cluster.cc:1428] Setting key 5f558369655a228187f033f2ba41f643
I20260812 14:32:06.055132 27735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320749e4d38f4a4ba6f9824d34218d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260812 14:32:06.055243 27735 leader_election.cc:302] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320749e4d38f4a4ba6f9824d34218d55; no voters:
I20260812 14:32:06.055426 27735 leader_election.cc:288] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:06.055467 27738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:06.055658 27738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [term 1 LEADER]: Becoming Leader. State: Replica: 320749e4d38f4a4ba6f9824d34218d55, State: Running, Role: LEADER
I20260812 14:32:06.055828 27738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320749e4d38f4a4ba6f9824d34218d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260812 14:32:06.055823 27735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:06.056437 27742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320749e4d38f4a4ba6f9824d34218d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320749e4d38f4a4ba6f9824d34218d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } } }
I20260812 14:32:06.056447 27741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320749e4d38f4a4ba6f9824d34218d55. Latest consensus state: current_term: 1 leader_uuid: "320749e4d38f4a4ba6f9824d34218d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320749e4d38f4a4ba6f9824d34218d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } } }
I20260812 14:32:06.056587 27742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:06.056597 27741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:06.056892 27748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:06.057425 27748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:06.059237 27748 catalog_manager.cc:1347] Generated new cluster ID: 16019db055884aa0a92e36bda755e534
I20260812 14:32:06.059295 27748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:06.061259 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:06.055517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:52075 (local address 127.0.0.1:33631)
0812 14:32:06.055770 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:06.055781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:06.055819 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:32:06.055948 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:06.055956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:06.056017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:06.056156 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:06.056590 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:06.058064 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:06.059086 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:06.059664 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:06.059802 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:06.060547 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:06.060551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:06.060551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:06.060772 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:06.060808 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:06.060823 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:06.060903 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:06.061015 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 14:32:06.061042 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260812 14:32:06.062363 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:06.067304 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:06.063796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:45051 (local address 127.0.0.1:33631)
0812 14:32:06.063834 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:06.063835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:06.063998 (+ 163us) server_negotiation.cc:407] Connection header received
0812 14:32:06.064031 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:06.064032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:06.064046 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:06.064116 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:06.064249 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:06.065013 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:06.065967 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:06.066280 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:06.066348 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:06.067070 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:06.067072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:06.067072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:06.067077 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:06.067093 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:06.067098 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:06.067159 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:06.067216 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:32:06.067229 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:32:06.084064 27748 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:06.084661 27748 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:06.089689 27748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320749e4d38f4a4ba6f9824d34218d55: Generated new TSK 0
I20260812 14:32:06.089864 27748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:06.097798 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27624
W20260812 14:32:06.114012 27596 connection.cc:535] client connection to 127.0.0.1:33631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33631): Connection reset by peer (error 104)
2026-08-12T14:32:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:06 dist-test-slave-1l3l krb5kdc[27787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:06 dist-test-slave-1l3l krb5kdc[27787](info): set up 2 sockets
Aug 12 14:32:06 dist-test-slave-1l3l krb5kdc[27787](info): commencing operation
krb5kdc: starting...
W20260812 14:32:07.292652 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:07 dist-test-slave-1l3l krb5kdc[27787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:07.416074 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:07.498230 27815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:07.498435 27815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:07.498458 27815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:07.499537 27815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:07.499573 27815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:07.499588 27815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:07.499598 27815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:07.499609 27815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:07.501060 27815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:07.501255 27815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:07.501433 27815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:07.506135 27821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:07.506181 27815 server_base.cc:1034] running on GCE node
W20260812 14:32:07.506143 27822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:07.506140 27829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:07.506563 27815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:07.506829 27815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:07.507967 27815 hybrid_clock.cc:648] HybridClock initialized: now 1786545127507951 us; error 32 us; skew 500 ppm
Aug 12 14:32:07 dist-test-slave-1l3l krb5kdc[27787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:07.513779 27815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:07.514205 27815 webserver.cc:460] Webserver started at http://127.0.0.1:41761/ using document root <none> and password file <none>
I20260812 14:32:07.514380 27815 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:07.514420 27815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:07.514514 27815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:07.515542 27815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "9a51fe128404491b883fb39f7043c019"
format_stamp: "Formatted at 2026-08-12 14:32:07 on dist-test-slave-1l3l"
server_key: "540275047889940131d7e7ae23563352"
server_key_iv: "b0fc8cd9750479eda3d76ef813c58793"
server_key_version: "encryptionkey@0"
I20260812 14:32:07.515884 27815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "9a51fe128404491b883fb39f7043c019"
format_stamp: "Formatted at 2026-08-12 14:32:07 on dist-test-slave-1l3l"
server_key: "540275047889940131d7e7ae23563352"
server_key_iv: "b0fc8cd9750479eda3d76ef813c58793"
server_key_version: "encryptionkey@0"
I20260812 14:32:07.517102 27815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:32:07.517805 27840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:07.517982 27815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:32:07.518079 27815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "9a51fe128404491b883fb39f7043c019"
format_stamp: "Formatted at 2026-08-12 14:32:07 on dist-test-slave-1l3l"
server_key: "540275047889940131d7e7ae23563352"
server_key_iv: "b0fc8cd9750479eda3d76ef813c58793"
server_key_version: "encryptionkey@0"
I20260812 14:32:07.518138 27815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:07.521425 27815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:07.523550 27815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:07.523660 27815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:07.527478 27815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44671
I20260812 14:32:07.527511 27937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44671 every 8 connection(s)
I20260812 14:32:07.527869 27815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:07.528863 27938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:07.531461 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 27815
I20260812 14:32:07.531613 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:07.531819 24247 external_mini_cluster.cc:1428] Setting key 7e285f2e52a3be2b1bfdcd84097c1978
I20260812 14:32:07.532140 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019: Bootstrap starting.
I20260812 14:32:07.532862 27938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:07.533197 27938 log.cc:824] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:07.533972 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019: No bootstrap required, opened a new log
I20260812 14:32:07.535065 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:07.532601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:38201 (local address 127.0.0.1:44671)
0812 14:32:07.532864 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:07.532875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:07.532914 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:32:07.533505 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:07.533516 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:07.533554 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:07.533711 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:07.534517 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:07.534521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:07.534521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:07.534526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:07.534573 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:07.534587 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:07.534657 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:07.534816 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 14:32:07.534860 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":102,"threads_started":1}
I20260812 14:32:07.536273 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:07.536473 27938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a51fe128404491b883fb39f7043c019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44671 } }
I20260812 14:32:07.536607 27938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:07.536640 27938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a51fe128404491b883fb39f7043c019, State: Initialized, Role: FOLLOWER
I20260812 14:32:07.536757 27938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a51fe128404491b883fb39f7043c019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44671 } }
I20260812 14:32:07.536849 27938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:07.536882 27938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:07.536929 27938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:07.537518 27938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a51fe128404491b883fb39f7043c019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44671 } }
I20260812 14:32:07.537628 27938 leader_election.cc:302] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a51fe128404491b883fb39f7043c019; no voters:
I20260812 14:32:07.538127 27938 leader_election.cc:288] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:07.538242 27948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:07.540340 27938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:07.541778 27948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [term 1 LEADER]: Becoming Leader. State: Replica: 9a51fe128404491b883fb39f7043c019, State: Running, Role: LEADER
I20260812 14:32:07.541927 27948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a51fe128404491b883fb39f7043c019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44671 } }
I20260812 14:32:07.543076 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a51fe128404491b883fb39f7043c019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a51fe128404491b883fb39f7043c019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44671 } } }
I20260812 14:32:07.543218 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:07.543401 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a51fe128404491b883fb39f7043c019. Latest consensus state: current_term: 1 leader_uuid: "9a51fe128404491b883fb39f7043c019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a51fe128404491b883fb39f7043c019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44671 } } }
I20260812 14:32:07.543507 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019 [sys.catalog]: This master's current role is: LEADER
W20260812 14:32:07.544543 27983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:07.544600 27983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:07.544649 27968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:07.544806 27968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:07.545034 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:07.543545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:52695 (local address 127.0.0.1:44671)
0812 14:32:07.543585 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:07.543588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:07.544077 (+ 489us) server_negotiation.cc:407] Connection header received
0812 14:32:07.544131 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:07.544132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:07.544138 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:07.544189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:07.544807 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:07.544808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:07.544809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:07.544813 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:07.544829 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:07.544834 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:07.544880 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:07.544955 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:32:07.544971 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:32:07.546379 27968 catalog_manager.cc:1347] Generated new cluster ID: a25b6777f2fb42de913fd77f1aadf7d3
I20260812 14:32:07.546434 27968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:07.555483 27968 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:07.555660 27968 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:07.562920 27968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a51fe128404491b883fb39f7043c019: Generated new TSK 0
I20260812 14:32:07.563102 27968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:07.567682 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 27815
2026-08-12T14:32:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:07 dist-test-slave-1l3l krb5kdc[27994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:07 dist-test-slave-1l3l krb5kdc[27994](info): set up 2 sockets
Aug 12 14:32:07 dist-test-slave-1l3l krb5kdc[27994](info): commencing operation
krb5kdc: starting...
W20260812 14:32:08.766948 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:08 dist-test-slave-1l3l krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:08.892433 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:08.974293 28019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:08.974500 28019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:08.974522 28019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:08.975739 28019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:08.975785 28019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:08.975798 28019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:08.975809 28019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:08.975821 28019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:08.977350 28019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:08.977586 28019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:08.977805 28019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:32:08.982813 28019 server_base.cc:1034] running on GCE node
W20260812 14:32:08.982781 28025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:08.982793 28032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:08.982920 28026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:08.983292 28019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:08.983594 28019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:08.984750 28019 hybrid_clock.cc:648] HybridClock initialized: now 1786545128984729 us; error 34 us; skew 500 ppm
Aug 12 14:32:08 dist-test-slave-1l3l krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:08.987203 28019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:08.987598 28019 webserver.cc:460] Webserver started at http://127.0.0.1:41697/ using document root <none> and password file <none>
I20260812 14:32:08.987828 28019 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:08.987882 28019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:08.987982 28019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:08.989001 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "56427bcfb4a74a628ac561098e6cf726"
format_stamp: "Formatted at 2026-08-12 14:32:08 on dist-test-slave-1l3l"
server_key: "293642fa5be02e77fbdbefdcd172b185"
server_key_iv: "280c2741903a3e8918ee2f7275308622"
server_key_version: "encryptionkey@0"
I20260812 14:32:08.989295 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "56427bcfb4a74a628ac561098e6cf726"
format_stamp: "Formatted at 2026-08-12 14:32:08 on dist-test-slave-1l3l"
server_key: "293642fa5be02e77fbdbefdcd172b185"
server_key_iv: "280c2741903a3e8918ee2f7275308622"
server_key_version: "encryptionkey@0"
I20260812 14:32:08.990502 28019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:32:08.991250 28044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:08.991410 28019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:08.991496 28019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "56427bcfb4a74a628ac561098e6cf726"
format_stamp: "Formatted at 2026-08-12 14:32:08 on dist-test-slave-1l3l"
server_key: "293642fa5be02e77fbdbefdcd172b185"
server_key_iv: "280c2741903a3e8918ee2f7275308622"
server_key_version: "encryptionkey@0"
I20260812 14:32:08.991569 28019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:09.006696 28019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:09.008706 28019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:09.008823 28019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:09.012691 28019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32791
I20260812 14:32:09.012740 28141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32791 every 8 connection(s)
I20260812 14:32:09.013051 28019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:09.013886 28142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:09.016179 28142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726: Bootstrap starting.
I20260812 14:32:09.016741 28142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:09.017011 28142 log.cc:824] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:09.017129 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28019
I20260812 14:32:09.017253 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:09.017468 24247 external_mini_cluster.cc:1428] Setting key 031c68d071ca045dd1f1c5f6fb589baf
I20260812 14:32:09.017896 28142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726: No bootstrap required, opened a new log
I20260812 14:32:09.020061 28142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56427bcfb4a74a628ac561098e6cf726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260812 14:32:09.020219 28142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:09.020244 28142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56427bcfb4a74a628ac561098e6cf726, State: Initialized, Role: FOLLOWER
I20260812 14:32:09.020356 28142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56427bcfb4a74a628ac561098e6cf726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260812 14:32:09.020421 28142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:09.020442 28142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:09.020486 28142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:09.021051 28142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56427bcfb4a74a628ac561098e6cf726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260812 14:32:09.021140 28142 leader_election.cc:302] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56427bcfb4a74a628ac561098e6cf726; no voters:
I20260812 14:32:09.021304 28142 leader_election.cc:288] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:09.021338 28153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:09.021503 28153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [term 1 LEADER]: Becoming Leader. State: Replica: 56427bcfb4a74a628ac561098e6cf726, State: Running, Role: LEADER
I20260812 14:32:09.021577 28142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:09.021642 28153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56427bcfb4a74a628ac561098e6cf726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260812 14:32:09.022049 28153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56427bcfb4a74a628ac561098e6cf726. Latest consensus state: current_term: 1 leader_uuid: "56427bcfb4a74a628ac561098e6cf726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56427bcfb4a74a628ac561098e6cf726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } } }
I20260812 14:32:09.022285 28153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:09.022693 28167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:09.022217 28154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56427bcfb4a74a628ac561098e6cf726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56427bcfb4a74a628ac561098e6cf726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } } }
I20260812 14:32:09.023129 28154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:09.023212 28167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:32:09 dist-test-slave-1l3l krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:32:09.028184 28167 catalog_manager.cc:1347] Generated new cluster ID: 8f3d1f7825d3499daa71034eb5271d71
I20260812 14:32:09.028247 28167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:09.029232 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:09.017953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:45203 (local address 127.0.0.1:32791)
0812 14:32:09.018124 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:09.018133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:09.018164 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:32:09.019104 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:09.019111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:09.019162 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:09.019276 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:09.020106 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:09.021405 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:09.022467 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:09.023071 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:09.023513 (+ 442us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:09.025912 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:09.025916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:09.025917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:09.025930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:09.027484 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:09.027924 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:09.027927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:09.027928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:09.028022 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:09.028327 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:09.028328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:09.028328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:09.028582 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:09.028739 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:09.028874 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:32:09.029065 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260812 14:32:09.030141 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:09.037916 28167 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:09.038475 28167 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:09.043972 28167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56427bcfb4a74a628ac561098e6cf726: Generated new TSK 0
I20260812 14:32:09.044168 28167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:09.044963 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:09.032177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:46517 (local address 127.0.0.1:32791)
0812 14:32:09.032215 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:09.032217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:09.037622 (+ 5405us) server_negotiation.cc:407] Connection header received
0812 14:32:09.037679 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:09.037680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:09.037692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:09.037759 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:09.038463 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:09.039300 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:09.040366 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:09.040683 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:09.040744 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:09.042179 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:09.042180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:09.042180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:09.042188 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:09.043764 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:09.044238 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:09.044240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:09.044240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:09.044306 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:09.044526 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:09.044526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:09.044527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:09.044629 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:09.044706 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:09.044843 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:32:09.044895 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:32:09.047653 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28019
W20260812 14:32:09.061858 27989 connection.cc:535] client connection to 127.0.0.1:32791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32791): Connection reset by peer (error 104)
2026-08-12T14:32:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:09 dist-test-slave-1l3l krb5kdc[28194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:09 dist-test-slave-1l3l krb5kdc[28194](info): set up 2 sockets
Aug 12 14:32:09 dist-test-slave-1l3l krb5kdc[28194](info): commencing operation
krb5kdc: starting...
W20260812 14:32:10.235476 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:10 dist-test-slave-1l3l krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:10.363381 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:10.446635 28216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:10.446862 28216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:10.446895 28216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:10.448112 28216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:10.448158 28216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:10.448176 28216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:10.448192 28216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:10.448206 28216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:10.449728 28216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:10.449988 28216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:10.450213 28216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:10.455222 28222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:10.455384 28216 server_base.cc:1034] running on GCE node
W20260812 14:32:10.455247 28223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:10.455235 28229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:10.455854 28216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:10.456168 28216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:10.457333 28216 hybrid_clock.cc:648] HybridClock initialized: now 1786545130457312 us; error 30 us; skew 500 ppm
Aug 12 14:32:10 dist-test-slave-1l3l krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:10.459836 28216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:10.460330 28216 webserver.cc:460] Webserver started at http://127.0.0.1:41177/ using document root <none> and password file <none>
I20260812 14:32:10.460546 28216 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:10.460597 28216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:10.460724 28216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:10.461809 28216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "b342dff5a1f2438d8253193370cb8848"
format_stamp: "Formatted at 2026-08-12 14:32:10 on dist-test-slave-1l3l"
server_key: "6a31ef37f68adc1fb11fd3ffe448601a"
server_key_iv: "861d590cc710bddab8bf06980f2cc448"
server_key_version: "encryptionkey@0"
I20260812 14:32:10.462167 28216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "b342dff5a1f2438d8253193370cb8848"
format_stamp: "Formatted at 2026-08-12 14:32:10 on dist-test-slave-1l3l"
server_key: "6a31ef37f68adc1fb11fd3ffe448601a"
server_key_iv: "861d590cc710bddab8bf06980f2cc448"
server_key_version: "encryptionkey@0"
I20260812 14:32:10.463574 28216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:32:10.464351 28245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:10.464504 28216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:10.464591 28216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "b342dff5a1f2438d8253193370cb8848"
format_stamp: "Formatted at 2026-08-12 14:32:10 on dist-test-slave-1l3l"
server_key: "6a31ef37f68adc1fb11fd3ffe448601a"
server_key_iv: "861d590cc710bddab8bf06980f2cc448"
server_key_version: "encryptionkey@0"
I20260812 14:32:10.464660 28216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:10.476581 28216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:10.478382 28216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:10.478523 28216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:10.482448 28216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42653
I20260812 14:32:10.482492 28335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42653 every 8 connection(s)
I20260812 14:32:10.482831 28216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:10.483762 28341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:10.486114 28341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848: Bootstrap starting.
I20260812 14:32:10.486661 28341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:10.486903 28341 log.cc:824] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:10.487589 28341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848: No bootstrap required, opened a new log
I20260812 14:32:10.489253 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28216
I20260812 14:32:10.489228 28341 raft_consensus.cc:348] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b342dff5a1f2438d8253193370cb8848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42653 } }
I20260812 14:32:10.489368 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:10.489380 28341 raft_consensus.cc:374] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:10.489423 28341 raft_consensus.cc:729] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b342dff5a1f2438d8253193370cb8848, State: Initialized, Role: FOLLOWER
I20260812 14:32:10.489583 24247 external_mini_cluster.cc:1428] Setting key 401bc51ddca0f6359b35f9d5ce624a30
I20260812 14:32:10.489564 28341 consensus_queue.cc:260] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b342dff5a1f2438d8253193370cb8848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42653 } }
I20260812 14:32:10.489622 28341 raft_consensus.cc:388] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:10.489646 28341 raft_consensus.cc:482] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:10.489691 28341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:10.490783 28341 raft_consensus.cc:504] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b342dff5a1f2438d8253193370cb8848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42653 } }
I20260812 14:32:10.490895 28341 leader_election.cc:302] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b342dff5a1f2438d8253193370cb8848; no voters:
I20260812 14:32:10.491060 28341 leader_election.cc:288] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:10.491083 28350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:10.491199 28350 raft_consensus.cc:686] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [term 1 LEADER]: Becoming Leader. State: Replica: b342dff5a1f2438d8253193370cb8848, State: Running, Role: LEADER
I20260812 14:32:10.491345 28341 sys_catalog.cc:564] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:10.491356 28350 consensus_queue.cc:237] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b342dff5a1f2438d8253193370cb8848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42653 } }
I20260812 14:32:10.491869 28352 sys_catalog.cc:455] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b342dff5a1f2438d8253193370cb8848. Latest consensus state: current_term: 1 leader_uuid: "b342dff5a1f2438d8253193370cb8848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b342dff5a1f2438d8253193370cb8848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42653 } } }
I20260812 14:32:10.491966 28352 sys_catalog.cc:458] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:10.492172 28351 sys_catalog.cc:455] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b342dff5a1f2438d8253193370cb8848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b342dff5a1f2438d8253193370cb8848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42653 } } }
I20260812 14:32:10.492537 28351 sys_catalog.cc:458] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:10.492714 28361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:10.493369 28361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:10.494933 28361 catalog_manager.cc:1347] Generated new cluster ID: efc8ffae711e4fe986fc05a25e67849d
I20260812 14:32:10.494985 28361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:10.495822 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:10.490108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:55563 (local address 127.0.0.1:42653)
0812 14:32:10.490331 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:10.490343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:10.490386 (+ 43us) server_negotiation.cc:407] Connection header received
0812 14:32:10.490516 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:10.490524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:10.490592 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:10.490746 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:10.491010 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:10.492437 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:10.493481 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:10.493913 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:10.494012 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:10.494975 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:10.494978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:10.494979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:10.495183 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:10.495215 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:10.495226 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:10.495298 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:10.495545 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 14:32:10.495576 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260812 14:32:10.496760 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:10.500870 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:10.498058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:34147 (local address 127.0.0.1:42653)
0812 14:32:10.498117 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:10.498119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:10.498227 (+ 108us) server_negotiation.cc:407] Connection header received
0812 14:32:10.498284 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:10.498285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:10.498305 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:10.498347 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:10.498517 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:10.498986 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:10.499666 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:10.499948 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:10.500005 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:10.500636 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:10.500638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:10.500638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:10.500642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:10.500657 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:10.500661 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:10.500710 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:10.500778 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 14:32:10.500790 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:32:10.502714 28361 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:10.503211 28361 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:10.511260 28361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b342dff5a1f2438d8253193370cb8848: Generated new TSK 0
I20260812 14:32:10.511438 28361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:10.514286 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28216
W20260812 14:32:10.530259 28191 connection.cc:535] client connection to 127.0.0.1:42653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42653): Connection reset by peer (error 104)
2026-08-12T14:32:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:10 dist-test-slave-1l3l krb5kdc[28388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:10 dist-test-slave-1l3l krb5kdc[28388](info): set up 2 sockets
Aug 12 14:32:10 dist-test-slave-1l3l krb5kdc[28388](info): commencing operation
krb5kdc: starting...
W20260812 14:32:11.696972 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:11 dist-test-slave-1l3l krb5kdc[28388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:32:11.827392 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:11.910095 28409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:11.910326 28409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:11.910359 28409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:11.911572 28409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:11.911619 28409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:11.911643 28409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:11.911659 28409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:11.911674 28409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:11.913266 28409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:11.913533 28409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:11.913763 28409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:11.918834 28423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:11.918841 28421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:11.918908 28409 server_base.cc:1034] running on GCE node
W20260812 14:32:11.918891 28420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:11.919279 28409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:11.919584 28409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:11.920737 28409 hybrid_clock.cc:648] HybridClock initialized: now 1786545131920716 us; error 30 us; skew 500 ppm
Aug 12 14:32:11 dist-test-slave-1l3l krb5kdc[28388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:11.923303 28409 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:32:11.923779 28409 webserver.cc:460] Webserver started at http://127.0.0.1:36163/ using document root <none> and password file <none>
I20260812 14:32:11.924015 28409 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:11.924067 28409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:11.924185 28409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:11.925289 28409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "8b98d6115cbd44df900e3b864c168244"
format_stamp: "Formatted at 2026-08-12 14:32:11 on dist-test-slave-1l3l"
server_key: "85b1d84fb10609b4489efad756ebab5e"
server_key_iv: "131d40f805908999df931dc481e4a6af"
server_key_version: "encryptionkey@0"
I20260812 14:32:11.925681 28409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "8b98d6115cbd44df900e3b864c168244"
format_stamp: "Formatted at 2026-08-12 14:32:11 on dist-test-slave-1l3l"
server_key: "85b1d84fb10609b4489efad756ebab5e"
server_key_iv: "131d40f805908999df931dc481e4a6af"
server_key_version: "encryptionkey@0"
I20260812 14:32:11.926999 28409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:32:11.927789 28435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:11.927960 28409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:32:11.928053 28409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "8b98d6115cbd44df900e3b864c168244"
format_stamp: "Formatted at 2026-08-12 14:32:11 on dist-test-slave-1l3l"
server_key: "85b1d84fb10609b4489efad756ebab5e"
server_key_iv: "131d40f805908999df931dc481e4a6af"
server_key_version: "encryptionkey@0"
I20260812 14:32:11.928119 28409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:11.931509 28409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:11.933924 28409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:11.934065 28409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:11.938551 28409 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33463
I20260812 14:32:11.938592 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33463 every 8 connection(s)
I20260812 14:32:11.939061 28409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:11.940549 28521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:11.942822 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244: Bootstrap starting.
I20260812 14:32:11.943151 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28409
I20260812 14:32:11.943252 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:11.943449 24247 external_mini_cluster.cc:1428] Setting key af9bf2659b2c239e62b4d0fd7cc18174
I20260812 14:32:11.943584 28521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:11.944033 28521 log.cc:824] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:11.944926 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244: No bootstrap required, opened a new log
I20260812 14:32:11.946211 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:11.943977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:60449 (local address 127.0.0.1:33463)
0812 14:32:11.944232 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:11.944243 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:11.944284 (+ 41us) server_negotiation.cc:407] Connection header received
0812 14:32:11.944847 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:11.944856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:11.944893 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:11.945070 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:11.945758 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:11.945762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:11.945763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:11.945767 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:11.945811 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:11.945825 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:11.945936 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:11.945968 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 14:32:11.946012 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260812 14:32:11.947244 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:11.947227 28521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b98d6115cbd44df900e3b864c168244" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260812 14:32:11.947410 28521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:11.947448 28521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b98d6115cbd44df900e3b864c168244, State: Initialized, Role: FOLLOWER
I20260812 14:32:11.947568 28521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b98d6115cbd44df900e3b864c168244" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260812 14:32:11.947650 28521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:11.947685 28521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:11.947770 28521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:11.948581 28521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b98d6115cbd44df900e3b864c168244" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260812 14:32:11.948714 28521 leader_election.cc:302] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b98d6115cbd44df900e3b864c168244; no voters:
I20260812 14:32:11.948884 28521 leader_election.cc:288] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:11.948937 28536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:11.949129 28536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [term 1 LEADER]: Becoming Leader. State: Replica: 8b98d6115cbd44df900e3b864c168244, State: Running, Role: LEADER
I20260812 14:32:11.949218 28521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:11.949258 28536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b98d6115cbd44df900e3b864c168244" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260812 14:32:11.949738 28540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b98d6115cbd44df900e3b864c168244. Latest consensus state: current_term: 1 leader_uuid: "8b98d6115cbd44df900e3b864c168244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b98d6115cbd44df900e3b864c168244" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } } }
I20260812 14:32:11.949843 28540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:11.949859 28539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b98d6115cbd44df900e3b864c168244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b98d6115cbd44df900e3b864c168244" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } } }
I20260812 14:32:11.949908 28539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:11.950683 28548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:11.951076 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:11.948971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:56349 (local address 127.0.0.1:33463)
0812 14:32:11.949010 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:11.949012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:11.949757 (+ 745us) server_negotiation.cc:407] Connection header received
0812 14:32:11.949869 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:11.949870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:11.949875 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:11.949905 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:11.950751 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:11.950753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:11.950754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:11.950758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:11.950774 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:11.950778 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:11.950827 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:11.951007 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 14:32:11.951018 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:32:11.951254 28548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:11.952914 28548 catalog_manager.cc:1347] Generated new cluster ID: b79818d756f74cefb0d2ca2212ce5114
I20260812 14:32:11.952971 28548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:11.967129 28548 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:11.967303 28548 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:11.970782 28548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b98d6115cbd44df900e3b864c168244: Generated new TSK 0
I20260812 14:32:11.970947 28548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:11.980779 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28409
2026-08-12T14:32:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:12 dist-test-slave-1l3l krb5kdc[28569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:12 dist-test-slave-1l3l krb5kdc[28569](info): set up 2 sockets
Aug 12 14:32:12 dist-test-slave-1l3l krb5kdc[28569](info): commencing operation
krb5kdc: starting...
W20260812 14:32:13.191421 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:13 dist-test-slave-1l3l krb5kdc[28569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:13.308280 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:39905
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:39905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:13.388804 28594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:13.389005 28594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:13.389035 28594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:13.390165 28594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:13.390208 28594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:13.390228 28594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:13.390242 28594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:13.390259 28594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:13.391772 28594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:39905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:39905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:13.392015 28594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:13.392225 28594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:13.396889 28609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:13.396927 28594 server_base.cc:1034] running on GCE node
W20260812 14:32:13.396942 28606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:13.396911 28604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:13.397392 28594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:13.397665 28594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:13.398792 28594 hybrid_clock.cc:648] HybridClock initialized: now 1786545133398784 us; error 28 us; skew 500 ppm
Aug 12 14:32:13 dist-test-slave-1l3l krb5kdc[28569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:13.404161 28594 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:13.404582 28594 webserver.cc:460] Webserver started at http://127.23.173.254:39519/ using document root <none> and password file <none>
I20260812 14:32:13.404793 28594 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:13.404842 28594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:13.404970 28594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:13.405983 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "3b4b7950df3d4fdbbe0f80a3862152b3"
format_stamp: "Formatted at 2026-08-12 14:32:13 on dist-test-slave-1l3l"
server_key: "b2a2d7d8f39130530229e54547a68ea9"
server_key_iv: "817e00af27b77f7db0991c155bdce1fe"
server_key_version: "encryptionkey@0"
I20260812 14:32:13.406332 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "3b4b7950df3d4fdbbe0f80a3862152b3"
format_stamp: "Formatted at 2026-08-12 14:32:13 on dist-test-slave-1l3l"
server_key: "b2a2d7d8f39130530229e54547a68ea9"
server_key_iv: "817e00af27b77f7db0991c155bdce1fe"
server_key_version: "encryptionkey@0"
I20260812 14:32:13.407584 28594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:32:13.408320 28621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:13.408488 28594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:13.408573 28594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "3b4b7950df3d4fdbbe0f80a3862152b3"
format_stamp: "Formatted at 2026-08-12 14:32:13 on dist-test-slave-1l3l"
server_key: "b2a2d7d8f39130530229e54547a68ea9"
server_key_iv: "817e00af27b77f7db0991c155bdce1fe"
server_key_version: "encryptionkey@0"
I20260812 14:32:13.408638 28594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:13.436345 28594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:13.438134 28594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:13.438277 28594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:13.442117 28594 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:39905
I20260812 14:32:13.442168 28704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:39905 every 8 connection(s)
I20260812 14:32:13.442498 28594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:13.442761 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28594
I20260812 14:32:13.442895 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:13.443109 24247 external_mini_cluster.cc:1428] Setting key 9888fdf2d9bb1a792803cf6f6d8ca483
I20260812 14:32:13.443511 28705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:13.446676 28705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3: Bootstrap starting.
I20260812 14:32:13.447365 28705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:13.447650 28705 log.cc:824] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:13.448400 28705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3: No bootstrap required, opened a new log
Aug 12 14:32:13 dist-test-slave-1l3l krb5kdc[28569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:13.450459 28705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39905 } }
I20260812 14:32:13.450593 28705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:13.450624 28705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4b7950df3d4fdbbe0f80a3862152b3, State: Initialized, Role: FOLLOWER
I20260812 14:32:13.450754 28705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39905 } }
I20260812 14:32:13.450838 28705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:13.450883 28705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:13.450927 28705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:13.451551 28705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39905 } }
I20260812 14:32:13.451651 28705 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4b7950df3d4fdbbe0f80a3862152b3; no voters:
I20260812 14:32:13.451822 28705 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:13.451867 28714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:13.452013 28714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4b7950df3d4fdbbe0f80a3862152b3, State: Running, Role: LEADER
I20260812 14:32:13.452113 28705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:13.452152 28714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39905 } }
I20260812 14:32:13.452541 28714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4b7950df3d4fdbbe0f80a3862152b3. Latest consensus state: current_term: 1 leader_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39905 } } }
I20260812 14:32:13.452615 28714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:13.452700 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4b7950df3d4fdbbe0f80a3862152b3" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 39905 } } }
I20260812 14:32:13.452749 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:13.453619 28728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:13.454133 28728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:13.454280 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:13.443606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40881 (local address 127.23.173.254:39905)
0812 14:32:13.443804 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:13.443814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:13.443842 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:32:13.444274 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:13.444281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:13.444327 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:13.444440 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:13.445281 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:13.446171 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:13.447207 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:13.447684 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:13.447805 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:13.450388 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:13.450392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:13.450392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:13.450402 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:13.452155 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:13.452575 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:13.452577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:13.452578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:13.452681 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:13.452965 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:13.452966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:13.452966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:13.453280 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:13.453444 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:13.453970 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 14:32:13.454075 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260812 14:32:13.455147 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:13.455731 28728 catalog_manager.cc:1347] Generated new cluster ID: 58268c063ec74405ba2856a91ebfa727
I20260812 14:32:13.455783 28728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:13.462028 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:13.456503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56323 (local address 127.23.173.254:39905)
0812 14:32:13.456536 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:13.456537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:13.456652 (+ 115us) server_negotiation.cc:407] Connection header received
0812 14:32:13.456710 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:13.456711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:13.456720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:13.456764 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:13.457230 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:13.457701 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:13.458359 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:13.458583 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:13.458637 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:13.459740 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:13.459741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:13.459741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:13.459748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:13.460994 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:13.461315 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:13.461316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:13.461316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:13.461368 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:13.461560 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:13.461561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:13.461561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:13.461663 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:13.461749 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:13.461885 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:32:13.461950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:32:13.468328 28728 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:13.469002 28728 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:13.476128 28728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4b7950df3d4fdbbe0f80a3862152b3: Generated new TSK 0
I20260812 14:32:13.476298 28728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:13.478184 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28594
W20260812 14:32:13.494549 28565 connection.cc:535] client connection to 127.23.173.254:39905 recv error: Network error: failed to read from TLS socket (remote: 127.23.173.254:39905): Connection reset by peer (error 104)
2026-08-12T14:32:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:13 dist-test-slave-1l3l krb5kdc[28757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:13 dist-test-slave-1l3l krb5kdc[28757](info): set up 2 sockets
Aug 12 14:32:13 dist-test-slave-1l3l krb5kdc[28757](info): commencing operation
krb5kdc: starting...
W20260812 14:32:14.655498 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:14 dist-test-slave-1l3l krb5kdc[28757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:14.776777 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:40217
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:40217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:14.859546 28778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:14.859793 28778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:14.859817 28778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:14.861007 28778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:14.861053 28778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:14.861068 28778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:14.861090 28778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:14.861107 28778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:14.862795 28778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:40217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:40217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:14.863046 28778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:14.863273 28778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:14.868049 28786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:14.868057 28787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:14.868073 28778 server_base.cc:1034] running on GCE node
W20260812 14:32:14.868067 28793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:14.868471 28778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:14.868748 28778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:14.869889 28778 hybrid_clock.cc:648] HybridClock initialized: now 1786545134869873 us; error 31 us; skew 500 ppm
Aug 12 14:32:14 dist-test-slave-1l3l krb5kdc[28757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545134, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:14.872498 28778 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:14.872922 28778 webserver.cc:460] Webserver started at http://127.23.173.254:34887/ using document root <none> and password file <none>
I20260812 14:32:14.873137 28778 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:14.873188 28778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:14.873315 28778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:14.874382 28778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "c727d085f884440f8b0b6bfed5af1782"
format_stamp: "Formatted at 2026-08-12 14:32:14 on dist-test-slave-1l3l"
server_key: "a135797e73904c18675f4514c74db6e2"
server_key_iv: "a60aff9706992ffd83da89be95198285"
server_key_version: "encryptionkey@0"
I20260812 14:32:14.874722 28778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "c727d085f884440f8b0b6bfed5af1782"
format_stamp: "Formatted at 2026-08-12 14:32:14 on dist-test-slave-1l3l"
server_key: "a135797e73904c18675f4514c74db6e2"
server_key_iv: "a60aff9706992ffd83da89be95198285"
server_key_version: "encryptionkey@0"
I20260812 14:32:14.876003 28778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:32:14.876665 28814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:14.876813 28778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:32:14.876899 28778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "c727d085f884440f8b0b6bfed5af1782"
format_stamp: "Formatted at 2026-08-12 14:32:14 on dist-test-slave-1l3l"
server_key: "a135797e73904c18675f4514c74db6e2"
server_key_iv: "a60aff9706992ffd83da89be95198285"
server_key_version: "encryptionkey@0"
I20260812 14:32:14.876959 28778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:14.891780 28778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:14.893461 28778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:14.893597 28778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:14.897435 28778 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:40217
I20260812 14:32:14.897468 28903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:40217 every 8 connection(s)
I20260812 14:32:14.897790 28778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:14.898597 28905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:14.900825 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782: Bootstrap starting.
I20260812 14:32:14.901336 28905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:14.901575 28905 log.cc:824] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:14.902069 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28778
I20260812 14:32:14.902186 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:14.902323 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782: No bootstrap required, opened a new log
I20260812 14:32:14.902369 24247 external_mini_cluster.cc:1428] Setting key 8b1f535459ba66324d756f3eed679cc8
I20260812 14:32:14.904788 28905 raft_consensus.cc:348] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c727d085f884440f8b0b6bfed5af1782" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40217 } }
I20260812 14:32:14.904928 28905 raft_consensus.cc:374] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:14.904964 28905 raft_consensus.cc:729] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c727d085f884440f8b0b6bfed5af1782, State: Initialized, Role: FOLLOWER
I20260812 14:32:14.905089 28905 consensus_queue.cc:260] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c727d085f884440f8b0b6bfed5af1782" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40217 } }
I20260812 14:32:14.905166 28905 raft_consensus.cc:388] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:14.905201 28905 raft_consensus.cc:482] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:14.905246 28905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:14.905936 28905 raft_consensus.cc:504] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c727d085f884440f8b0b6bfed5af1782" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40217 } }
I20260812 14:32:14.906036 28905 leader_election.cc:302] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c727d085f884440f8b0b6bfed5af1782; no voters:
I20260812 14:32:14.906199 28905 leader_election.cc:288] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:14.906234 28912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:14.906395 28912 raft_consensus.cc:686] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [term 1 LEADER]: Becoming Leader. State: Replica: c727d085f884440f8b0b6bfed5af1782, State: Running, Role: LEADER
I20260812 14:32:14.906461 28905 sys_catalog.cc:564] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:14.906566 28912 consensus_queue.cc:237] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c727d085f884440f8b0b6bfed5af1782" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40217 } }
I20260812 14:32:14.906952 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c727d085f884440f8b0b6bfed5af1782. Latest consensus state: current_term: 1 leader_uuid: "c727d085f884440f8b0b6bfed5af1782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c727d085f884440f8b0b6bfed5af1782" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40217 } } }
I20260812 14:32:14.906996 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c727d085f884440f8b0b6bfed5af1782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c727d085f884440f8b0b6bfed5af1782" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 40217 } } }
I20260812 14:32:14.907119 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:14.907335 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:14.907820 28925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:14.908305 28925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:14.908591 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:14.902854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45611 (local address 127.23.173.254:40217)
0812 14:32:14.903034 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:14.903043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:14.903069 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:32:14.903579 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:14.903586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:14.903634 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:14.903790 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:14.903999 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:14.904836 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:14.905950 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:14.906522 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:14.906658 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:14.907790 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:14.907796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:14.907796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:14.908005 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:14.908044 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:14.908058 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:14.908133 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:14.908385 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 14:32:14.908414 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260812 14:32:14.909772 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:14.909870 28925 catalog_manager.cc:1347] Generated new cluster ID: d87fc5907a204f89bb420988bdf5c314
I20260812 14:32:14.909920 28925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:14.914829 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:14.912087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47813 (local address 127.23.173.254:40217)
0812 14:32:14.912123 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:14.912125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:14.912283 (+ 158us) server_negotiation.cc:407] Connection header received
0812 14:32:14.912321 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:14.912322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:14.912331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:14.912372 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:14.912491 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:14.912950 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:14.913646 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:14.913864 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:14.913921 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:14.914587 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:14.914588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:14.914588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:14.914593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:14.914608 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:14.914612 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:14.914661 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:14.914737 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:32:14.914749 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:32:14.919843 28925 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:14.920351 28925 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:14.923967 28925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c727d085f884440f8b0b6bfed5af1782: Generated new TSK 0
I20260812 14:32:14.924125 28925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:14.926167 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28778
W20260812 14:32:14.942126 28754 connection.cc:535] client connection to 127.23.173.254:40217 recv error: Network error: failed to read from TLS socket (remote: 127.23.173.254:40217): Connection reset by peer (error 104)
2026-08-12T14:32:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:14 dist-test-slave-1l3l krb5kdc[28951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:14 dist-test-slave-1l3l krb5kdc[28951](info): set up 2 sockets
Aug 12 14:32:14 dist-test-slave-1l3l krb5kdc[28951](info): commencing operation
krb5kdc: starting...
W20260812 14:32:16.124239 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:16 dist-test-slave-1l3l krb5kdc[28951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:16.244556 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35193
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:16.326452 28978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:16.326676 28978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:16.326709 28978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:16.327927 28978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:16.327970 28978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:16.327988 28978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:16.328004 28978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:16.328020 28978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:16.329566 28978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:16.329823 28978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:16.330044 28978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:16.334829 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:16.334824 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:16.334880 28990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:16.334986 28978 server_base.cc:1034] running on GCE node
I20260812 14:32:16.335300 28978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:16.335625 28978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:16.336771 28978 hybrid_clock.cc:648] HybridClock initialized: now 1786545136336762 us; error 48 us; skew 500 ppm
Aug 12 14:32:16 dist-test-slave-1l3l krb5kdc[28951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545136, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:16.339216 28978 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:16.339639 28978 webserver.cc:460] Webserver started at http://127.23.173.254:40647/ using document root <none> and password file <none>
I20260812 14:32:16.339879 28978 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:16.339931 28978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:16.340067 28978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:16.341111 28978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "bb73dd053964486ba095b995bb263595"
format_stamp: "Formatted at 2026-08-12 14:32:16 on dist-test-slave-1l3l"
server_key: "692f9a9cd508791695bda1f7f77f5ce4"
server_key_iv: "ebf5de6e07eebfcfa33615fd03886e1e"
server_key_version: "encryptionkey@0"
I20260812 14:32:16.341461 28978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "bb73dd053964486ba095b995bb263595"
format_stamp: "Formatted at 2026-08-12 14:32:16 on dist-test-slave-1l3l"
server_key: "692f9a9cd508791695bda1f7f77f5ce4"
server_key_iv: "ebf5de6e07eebfcfa33615fd03886e1e"
server_key_version: "encryptionkey@0"
I20260812 14:32:16.342732 28978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:32:16.343428 29000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:16.343596 28978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:32:16.343681 28978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "bb73dd053964486ba095b995bb263595"
format_stamp: "Formatted at 2026-08-12 14:32:16 on dist-test-slave-1l3l"
server_key: "692f9a9cd508791695bda1f7f77f5ce4"
server_key_iv: "ebf5de6e07eebfcfa33615fd03886e1e"
server_key_version: "encryptionkey@0"
I20260812 14:32:16.343781 28978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:16.347050 28978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:16.349270 28978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:16.349407 28978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:16.353292 28978 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:35193
I20260812 14:32:16.353346 29095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:35193 every 8 connection(s)
I20260812 14:32:16.353657 28978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:16.354601 29096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:16.356914 29096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595: Bootstrap starting.
I20260812 14:32:16.357473 29096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:16.357730 29096 log.cc:824] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:16.358398 29096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595: No bootstrap required, opened a new log
I20260812 14:32:16.359282 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 28978
I20260812 14:32:16.359411 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:16.359608 24247 external_mini_cluster.cc:1428] Setting key 4305b0b6ff22533cbf978bdddd5576ce
I20260812 14:32:16.360361 29096 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73dd053964486ba095b995bb263595" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35193 } }
I20260812 14:32:16.360500 29096 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:16.360522 29096 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb73dd053964486ba095b995bb263595, State: Initialized, Role: FOLLOWER
I20260812 14:32:16.360633 29096 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73dd053964486ba095b995bb263595" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35193 } }
I20260812 14:32:16.360695 29096 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:16.360715 29096 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:16.360760 29096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:16.361589 29096 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73dd053964486ba095b995bb263595" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35193 } }
I20260812 14:32:16.361689 29096 leader_election.cc:302] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb73dd053964486ba095b995bb263595; no voters:
I20260812 14:32:16.361851 29096 leader_election.cc:288] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:16.361881 29108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:16.362046 29108 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [term 1 LEADER]: Becoming Leader. State: Replica: bb73dd053964486ba095b995bb263595, State: Running, Role: LEADER
I20260812 14:32:16.362066 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:16.360159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59385 (local address 127.23.173.254:35193)
0812 14:32:16.360352 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:16.360361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:16.360386 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:32:16.360481 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:16.360486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:16.360514 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:16.360678 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:16.361533 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:16.361537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:16.361537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:16.361541 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:16.361585 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:16.361596 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:16.361652 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:16.361756 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 14:32:16.361803 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260812 14:32:16.362092 29096 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:16.362187 29108 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73dd053964486ba095b995bb263595" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35193 } }
I20260812 14:32:16.362526 29109 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb73dd053964486ba095b995bb263595. Latest consensus state: current_term: 1 leader_uuid: "bb73dd053964486ba095b995bb263595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73dd053964486ba095b995bb263595" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35193 } } }
I20260812 14:32:16.362608 29109 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:16.362905 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb73dd053964486ba095b995bb263595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73dd053964486ba095b995bb263595" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35193 } } }
I20260812 14:32:16.362986 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:16.363251 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 14:32:16.363940 29131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:16.363996 29131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:16.364106 29120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:16.364275 29120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:16.365787 29120 catalog_manager.cc:1347] Generated new cluster ID: c7e39dfef0ab45468d0f5913691d6118
I20260812 14:32:16.365831 29120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:16.372849 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:16.368000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60497 (local address 127.23.173.254:35193)
0812 14:32:16.368036 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:16.368038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:16.371866 (+ 3828us) server_negotiation.cc:407] Connection header received
0812 14:32:16.371964 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:16.371965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:16.371972 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:16.372029 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:16.372631 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:16.372632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:16.372633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:16.372637 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:16.372654 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:16.372659 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:16.372718 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:16.372755 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:32:16.372770 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:32:16.381847 29120 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:16.382004 29120 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:16.388952 29120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb73dd053964486ba095b995bb263595: Generated new TSK 0
I20260812 14:32:16.389135 29120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:16.398355 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 28978
2026-08-12T14:32:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:16 dist-test-slave-1l3l krb5kdc[29148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:16 dist-test-slave-1l3l krb5kdc[29148](info): set up 2 sockets
Aug 12 14:32:16 dist-test-slave-1l3l krb5kdc[29148](info): commencing operation
krb5kdc: starting...
W20260812 14:32:17.584147 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:17 dist-test-slave-1l3l krb5kdc[29148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:17.696261 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:36887
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:17.776083 29177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:17.776278 29177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:17.776310 29177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:17.777427 29177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:17.777468 29177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:17.777486 29177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:17.777503 29177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:17.777518 29177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:17.778968 29177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:17.779213 29177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:17.779418 29177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:17.784040 29190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:17.784188 29177 server_base.cc:1034] running on GCE node
W20260812 14:32:17.784070 29186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:17.784098 29187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:17.784564 29177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:17.784830 29177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:17.785975 29177 hybrid_clock.cc:648] HybridClock initialized: now 1786545137785962 us; error 31 us; skew 500 ppm
Aug 12 14:32:17 dist-test-slave-1l3l krb5kdc[29148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:17.788383 29177 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:17.788787 29177 webserver.cc:460] Webserver started at http://127.23.173.254:43271/ using document root <none> and password file <none>
I20260812 14:32:17.788973 29177 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:17.789022 29177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:17.789144 29177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:17.790138 29177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "a0827c924e614746a452b9d01fd54946"
format_stamp: "Formatted at 2026-08-12 14:32:17 on dist-test-slave-1l3l"
server_key: "1c3818f435735e9bd6bd1749d94b0872"
server_key_iv: "a01d6b231c7dd02ec64dc8dbf5cc570c"
server_key_version: "encryptionkey@0"
I20260812 14:32:17.790463 29177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "a0827c924e614746a452b9d01fd54946"
format_stamp: "Formatted at 2026-08-12 14:32:17 on dist-test-slave-1l3l"
server_key: "1c3818f435735e9bd6bd1749d94b0872"
server_key_iv: "a01d6b231c7dd02ec64dc8dbf5cc570c"
server_key_version: "encryptionkey@0"
I20260812 14:32:17.791697 29177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:32:17.792421 29203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:17.792605 29177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:17.792688 29177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "a0827c924e614746a452b9d01fd54946"
format_stamp: "Formatted at 2026-08-12 14:32:17 on dist-test-slave-1l3l"
server_key: "1c3818f435735e9bd6bd1749d94b0872"
server_key_iv: "a01d6b231c7dd02ec64dc8dbf5cc570c"
server_key_version: "encryptionkey@0"
I20260812 14:32:17.792752 29177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:17.804205 29177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:17.806185 29177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:17.806318 29177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:17.810143 29177 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:36887
I20260812 14:32:17.810180 29293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:36887 every 8 connection(s)
I20260812 14:32:17.810497 29177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:17.811285 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:17.813596 29294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946: Bootstrap starting.
I20260812 14:32:17.814114 29294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:17.814344 29294 log.cc:824] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:17.814958 29294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946: No bootstrap required, opened a new log
I20260812 14:32:17.816555 29294 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0827c924e614746a452b9d01fd54946" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36887 } }
I20260812 14:32:17.816687 29294 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:17.816708 29294 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0827c924e614746a452b9d01fd54946, State: Initialized, Role: FOLLOWER
I20260812 14:32:17.816816 29294 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0827c924e614746a452b9d01fd54946" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36887 } }
I20260812 14:32:17.816890 29294 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:17.816915 29294 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:17.816946 29294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:17.817725 29294 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0827c924e614746a452b9d01fd54946" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36887 } }
I20260812 14:32:17.817812 29294 leader_election.cc:302] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0827c924e614746a452b9d01fd54946; no voters:
I20260812 14:32:17.817929 29294 leader_election.cc:288] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:17.817973 29298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:17.818150 29294 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:17.818172 29298 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [term 1 LEADER]: Becoming Leader. State: Replica: a0827c924e614746a452b9d01fd54946, State: Running, Role: LEADER
I20260812 14:32:17.818321 29298 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0827c924e614746a452b9d01fd54946" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36887 } }
I20260812 14:32:17.818684 29299 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0827c924e614746a452b9d01fd54946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0827c924e614746a452b9d01fd54946" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36887 } } }
I20260812 14:32:17.818768 29299 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:17.818928 29303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0827c924e614746a452b9d01fd54946. Latest consensus state: current_term: 1 leader_uuid: "a0827c924e614746a452b9d01fd54946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0827c924e614746a452b9d01fd54946" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 36887 } } }
I20260812 14:32:17.818994 29303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:17.819269 29310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:17.819758 29310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:17.820514 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29177
I20260812 14:32:17.820631 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:17.820827 24247 external_mini_cluster.cc:1428] Setting key 361232de1f5974b1fc973d63f3612258
I20260812 14:32:17.821237 29310 catalog_manager.cc:1347] Generated new cluster ID: eaaa67575da14fe6b85fb3a701acb95a
I20260812 14:32:17.821281 29310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:32:17 dist-test-slave-1l3l krb5kdc[29148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:17.834008 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:17.821231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55521 (local address 127.23.173.254:36887)
0812 14:32:17.823819 (+ 2588us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:17.823829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:17.823864 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:32:17.823981 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:17.823988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:17.824057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:17.824203 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:17.824880 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:17.826295 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:17.827294 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:17.827942 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:17.828051 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:17.830690 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:17.830694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:17.830694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:17.830706 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:17.832522 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:17.832894 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:17.832897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:17.832897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:17.832994 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:17.833178 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:17.833178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:17.833178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:17.833445 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:17.833559 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:17.833670 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 14:32:17.833796 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2538,"thread_start_us":70,"threads_started":1}
I20260812 14:32:17.834868 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:17.846673 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:17.839957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57633 (local address 127.23.173.254:36887)
0812 14:32:17.839993 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:17.839995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:17.840171 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:32:17.840203 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:17.840203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:17.840214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:17.840275 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:17.840850 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:17.841606 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:17.842517 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:17.842837 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:17.842905 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:17.844155 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:17.844156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:17.844157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:17.844163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:17.845642 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:17.845980 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:17.845981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:17.845981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:17.846039 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:17.846204 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:17.846205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:17.846205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:17.846324 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:17.846409 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:17.846509 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:32:17.846585 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:32:17.850230 29310 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:17.850847 29310 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:17.860359 29310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0827c924e614746a452b9d01fd54946: Generated new TSK 0
I20260812 14:32:17.860524 29310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:17.863155 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29177
2026-08-12T14:32:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:17 dist-test-slave-1l3l krb5kdc[29342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:17 dist-test-slave-1l3l krb5kdc[29342](info): set up 2 sockets
Aug 12 14:32:17 dist-test-slave-1l3l krb5kdc[29342](info): commencing operation
krb5kdc: starting...
W20260812 14:32:19.025440 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:19 dist-test-slave-1l3l krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:19.138041 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35955
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:35955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:19.218250 29363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:19.218451 29363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:19.218483 29363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:19.219599 29363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:19.219640 29363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:19.219666 29363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:19.219681 29363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:19.219696 29363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:19.221158 29363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:35955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:35955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:19.221405 29363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:19.221603 29363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:19.226297 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:19.226274 29375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:19.226325 29363 server_base.cc:1034] running on GCE node
W20260812 14:32:19.226274 29377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:19.226790 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:19.227061 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:19.228201 29363 hybrid_clock.cc:648] HybridClock initialized: now 1786545139228186 us; error 31 us; skew 500 ppm
Aug 12 14:32:19 dist-test-slave-1l3l krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545139, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:19.233564 29363 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:19.234012 29363 webserver.cc:460] Webserver started at http://127.23.173.254:40807/ using document root <none> and password file <none>
I20260812 14:32:19.234203 29363 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:19.234251 29363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:19.234372 29363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:19.235394 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "08cc53f099d640aa934cf902399c1ab4"
format_stamp: "Formatted at 2026-08-12 14:32:19 on dist-test-slave-1l3l"
server_key: "e294d95060a0b9eaf52626888ea3f44d"
server_key_iv: "5eebcf4cd8a20416a82674e1eca18230"
server_key_version: "encryptionkey@0"
I20260812 14:32:19.235759 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "08cc53f099d640aa934cf902399c1ab4"
format_stamp: "Formatted at 2026-08-12 14:32:19 on dist-test-slave-1l3l"
server_key: "e294d95060a0b9eaf52626888ea3f44d"
server_key_iv: "5eebcf4cd8a20416a82674e1eca18230"
server_key_version: "encryptionkey@0"
I20260812 14:32:19.236980 29363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:19.237630 29388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:19.237772 29363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:19.237854 29363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "08cc53f099d640aa934cf902399c1ab4"
format_stamp: "Formatted at 2026-08-12 14:32:19 on dist-test-slave-1l3l"
server_key: "e294d95060a0b9eaf52626888ea3f44d"
server_key_iv: "5eebcf4cd8a20416a82674e1eca18230"
server_key_version: "encryptionkey@0"
I20260812 14:32:19.237933 29363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:19.257095 29363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:19.259207 29363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:19.259341 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:19.263178 29363 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:35955
I20260812 14:32:19.263211 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:35955 every 8 connection(s)
I20260812 14:32:19.263512 29363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:19.264396 29471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:19.266776 29471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4: Bootstrap starting.
I20260812 14:32:19.267298 29471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:19.267556 29471 log.cc:824] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:19.268220 29471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4: No bootstrap required, opened a new log
I20260812 14:32:19.269735 29471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cc53f099d640aa934cf902399c1ab4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35955 } }
I20260812 14:32:19.269865 29471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:19.269886 29471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cc53f099d640aa934cf902399c1ab4, State: Initialized, Role: FOLLOWER
I20260812 14:32:19.270008 29471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cc53f099d640aa934cf902399c1ab4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35955 } }
I20260812 14:32:19.270069 29471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:19.270088 29471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:19.270121 29471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:19.270925 29471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cc53f099d640aa934cf902399c1ab4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35955 } }
I20260812 14:32:19.271016 29471 leader_election.cc:302] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cc53f099d640aa934cf902399c1ab4; no voters:
I20260812 14:32:19.271160 29471 leader_election.cc:288] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:19.271204 29475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:19.271349 29475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 08cc53f099d640aa934cf902399c1ab4, State: Running, Role: LEADER
I20260812 14:32:19.271399 29471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:19.271440 29475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cc53f099d640aa934cf902399c1ab4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35955 } }
I20260812 14:32:19.271854 29478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08cc53f099d640aa934cf902399c1ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cc53f099d640aa934cf902399c1ab4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35955 } } }
I20260812 14:32:19.271955 29478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:19.272126 29480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cc53f099d640aa934cf902399c1ab4. Latest consensus state: current_term: 1 leader_uuid: "08cc53f099d640aa934cf902399c1ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cc53f099d640aa934cf902399c1ab4" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 35955 } } }
I20260812 14:32:19.272193 29480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:19.272511 29491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:19.272635 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29363
I20260812 14:32:19.272743 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:19.272943 24247 external_mini_cluster.cc:1428] Setting key c8bef37a4a8a93c0df0c0ca2a489de67
I20260812 14:32:19.273084 29491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:19.274730 29491 catalog_manager.cc:1347] Generated new cluster ID: d5774b74cf634e73a1a12447b7c110b2
I20260812 14:32:19.274780 29491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:19.278228 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:19.273383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39701 (local address 127.23.173.254:35955)
0812 14:32:19.273872 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:19.273880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:19.273906 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:32:19.273992 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:19.273997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:19.274064 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:19.274190 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:19.274510 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:19.275733 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:19.276497 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:19.276917 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:19.277019 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:19.277648 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:19.277653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:19.277653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:19.277813 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:19.277843 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:19.277853 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:19.277909 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:19.277984 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:32:19.278055 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":78,"threads_started":1}
I20260812 14:32:19.279043 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:19.280171 29491 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:19.280833 29491 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:19.285244 29491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08cc53f099d640aa934cf902399c1ab4: Generated new TSK 0
I20260812 14:32:19.285405 29491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:19.286873 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:19.284063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58815 (local address 127.23.173.254:35955)
0812 14:32:19.284105 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:19.284107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:19.284301 (+ 194us) server_negotiation.cc:407] Connection header received
0812 14:32:19.284355 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:19.284356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:19.284366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:19.284417 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:19.284536 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:19.285079 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:19.285862 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:19.286072 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:19.286125 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:19.286667 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:19.286668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:19.286669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:19.286673 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:19.286686 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:19.286690 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:19.286733 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:19.286785 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:32:19.286809 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:32:19.288712 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29363
2026-08-12T14:32:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:19 dist-test-slave-1l3l krb5kdc[29520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:19 dist-test-slave-1l3l krb5kdc[29520](info): set up 2 sockets
Aug 12 14:32:19 dist-test-slave-1l3l krb5kdc[29520](info): commencing operation
krb5kdc: starting...
W20260812 14:32:20.475585 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:20 dist-test-slave-1l3l krb5kdc[29520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:20.603644 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43009
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:20.686852 29542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:20.687073 29542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:20.687098 29542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:20.688230 29542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:20.688267 29542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:20.688280 29542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:20.688292 29542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:20.688301 29542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:20.689800 29542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:43009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:20.690009 29542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:20.690187 29542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:20.695036 29554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:20.695053 29559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:20.695075 29556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:20.695080 29542 server_base.cc:1034] running on GCE node
I20260812 14:32:20.695474 29542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:20.695822 29542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:20.696996 29542 hybrid_clock.cc:648] HybridClock initialized: now 1786545140696972 us; error 36 us; skew 500 ppm
Aug 12 14:32:20 dist-test-slave-1l3l krb5kdc[29520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:20.699611 29542 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:20.700088 29542 webserver.cc:460] Webserver started at http://127.23.173.254:38033/ using document root <none> and password file <none>
I20260812 14:32:20.700289 29542 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:20.700331 29542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:20.700474 29542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:20.701555 29542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "9550325c64fd446aae1a4f8e6770d00b"
format_stamp: "Formatted at 2026-08-12 14:32:20 on dist-test-slave-1l3l"
server_key: "010addca6c3ce5bf1df8bb9ee4373f10"
server_key_iv: "74f4d5b06f47a9fa59f2b3b0a80e2a61"
server_key_version: "encryptionkey@0"
I20260812 14:32:20.701898 29542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "9550325c64fd446aae1a4f8e6770d00b"
format_stamp: "Formatted at 2026-08-12 14:32:20 on dist-test-slave-1l3l"
server_key: "010addca6c3ce5bf1df8bb9ee4373f10"
server_key_iv: "74f4d5b06f47a9fa59f2b3b0a80e2a61"
server_key_version: "encryptionkey@0"
I20260812 14:32:20.703156 29542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:32:20.703873 29571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:20.704056 29542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:20.704141 29542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "9550325c64fd446aae1a4f8e6770d00b"
format_stamp: "Formatted at 2026-08-12 14:32:20 on dist-test-slave-1l3l"
server_key: "010addca6c3ce5bf1df8bb9ee4373f10"
server_key_iv: "74f4d5b06f47a9fa59f2b3b0a80e2a61"
server_key_version: "encryptionkey@0"
I20260812 14:32:20.704197 29542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:20.707381 29542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:20.709302 29542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:20.709412 29542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:20.713286 29542 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:43009
I20260812 14:32:20.713335 29655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:43009 every 8 connection(s)
I20260812 14:32:20.713675 29542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:20.714685 29656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:20.717100 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b: Bootstrap starting.
I20260812 14:32:20.717657 29656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:20.717934 29656 log.cc:824] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:20.718613 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b: No bootstrap required, opened a new log
I20260812 14:32:20.719148 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29542
I20260812 14:32:20.719264 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:20.719453 24247 external_mini_cluster.cc:1428] Setting key 2b20f7e04616cf9537d291b4ce1d153a
I20260812 14:32:20.720530 29656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9550325c64fd446aae1a4f8e6770d00b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43009 } }
I20260812 14:32:20.720705 29656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:32:20.720744 29656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9550325c64fd446aae1a4f8e6770d00b, State: Initialized, Role: FOLLOWER
I20260812 14:32:20.720888 29656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9550325c64fd446aae1a4f8e6770d00b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43009 } }
I20260812 14:32:20.721215 29656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:32:20.721258 29656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:32:20.721307 29656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:20.721763 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:20.719964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59019 (local address 127.23.173.254:43009)
0812 14:32:20.720147 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:20.720157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:20.720214 (+ 57us) server_negotiation.cc:407] Connection header received
0812 14:32:20.720497 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:20.720505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:20.720553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:20.720718 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:20.721342 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:20.721346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:20.721346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:20.721351 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:20.721400 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:20.721415 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:20.721475 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:20.721569 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 14:32:20.721610 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260812 14:32:20.721913 29656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9550325c64fd446aae1a4f8e6770d00b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43009 } }
I20260812 14:32:20.722031 29656 leader_election.cc:302] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9550325c64fd446aae1a4f8e6770d00b; no voters:
I20260812 14:32:20.722193 29656 leader_election.cc:288] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:32:20.722234 29664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:20.722350 29664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [term 1 LEADER]: Becoming Leader. State: Replica: 9550325c64fd446aae1a4f8e6770d00b, State: Running, Role: LEADER
I20260812 14:32:20.722438 29656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:20.722426 29664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9550325c64fd446aae1a4f8e6770d00b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43009 } }
I20260812 14:32:20.722667 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:20.722821 29664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9550325c64fd446aae1a4f8e6770d00b. Latest consensus state: current_term: 1 leader_uuid: "9550325c64fd446aae1a4f8e6770d00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9550325c64fd446aae1a4f8e6770d00b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43009 } } }
I20260812 14:32:20.722914 29664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:20.722833 29666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9550325c64fd446aae1a4f8e6770d00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9550325c64fd446aae1a4f8e6770d00b" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 43009 } } }
I20260812 14:32:20.723055 29666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b [sys.catalog]: This master's current role is: LEADER
W20260812 14:32:20.724269 29695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:20.724327 29695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:20.724447 29676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:20.724649 29676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:20.725507 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:20.724242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56091 (local address 127.23.173.254:43009)
0812 14:32:20.724283 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:20.724284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:20.724530 (+ 246us) server_negotiation.cc:407] Connection header received
0812 14:32:20.724632 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:20.724632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:20.724637 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:20.724666 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:20.725301 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:20.725302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:20.725303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:32:20.725307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:20.725321 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:32:20.725326 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:20.725351 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:20.725438 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:32:20.725449 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:32:20.726148 29676 catalog_manager.cc:1347] Generated new cluster ID: bc9c5d0d8b194a66a46c301a931a7f1e
I20260812 14:32:20.726195 29676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:20.735999 29676 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:20.736152 29676 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:20.739981 29676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9550325c64fd446aae1a4f8e6770d00b: Generated new TSK 0
I20260812 14:32:20.740232 29676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:32:20.750546 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29542
2026-08-12T14:32:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1454 ms)
[----------] 18 tests from AuthTokenIssuingTest (26404 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:20 dist-test-slave-1l3l krb5kdc[29709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:20 dist-test-slave-1l3l krb5kdc[29709](info): set up 2 sockets
Aug 12 14:32:20 dist-test-slave-1l3l krb5kdc[29709](info): commencing operation
krb5kdc: starting...
W20260812 14:32:21.919579 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:21 dist-test-slave-1l3l krb5kdc[29709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:22.036418 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46821
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46821,127.23.173.253:37975,127.23.173.252:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:22.115581 29731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:32:22.115823 29731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:22.115854 29731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:22.115885 29731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:22.117017 29731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:22.117056 29731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:22.117075 29731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:22.117091 29731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:22.117105 29731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:22.118530 29731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46821,127.23.173.253:37975,127.23.173.252:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:22.118767 29731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:22.118965 29731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:32:22.123598 29731 server_base.cc:1034] running on GCE node
W20260812 14:32:22.123572 29742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:22.123521 29741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:22.123544 29747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:22.124060 29731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:22.124341 29731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:22.125486 29731 hybrid_clock.cc:648] HybridClock initialized: now 1786545142125470 us; error 39 us; skew 500 ppm
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:22.127933 29731 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:22.128368 29731 webserver.cc:460] Webserver started at http://127.23.173.254:37105/ using document root <none> and password file <none>
I20260812 14:32:22.128609 29731 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:22.128718 29731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:22.128875 29731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:22.129916 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "4f06bf2f88884c5283e1befea4e5ac6e"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "21e017db0e79a0198e7649ed8f8fedfb"
server_key_iv: "c3a1a8f01b843113cad85f194ab389e6"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.130265 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "4f06bf2f88884c5283e1befea4e5ac6e"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "21e017db0e79a0198e7649ed8f8fedfb"
server_key_iv: "c3a1a8f01b843113cad85f194ab389e6"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.131537 29731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:32:22.132283 29759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:22.132433 29731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:22.132508 29731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "4f06bf2f88884c5283e1befea4e5ac6e"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "21e017db0e79a0198e7649ed8f8fedfb"
server_key_iv: "c3a1a8f01b843113cad85f194ab389e6"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.132571 29731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:22.146816 29731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:22.148557 29731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:22.148690 29731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:22.152714 29731 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:46821
I20260812 14:32:22.152747 29838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:46821 every 8 connection(s)
I20260812 14:32:22.153080 29731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:22.153944 29840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:22.155431 29840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:22.160904 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29731
I20260812 14:32:22.161028 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:22.161199 24247 external_mini_cluster.cc:1428] Setting key 0bca3df124538a33a45c63c7a5a5c7d1
I20260812 14:32:22.164340 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.156055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46821 (local address 127.0.0.1:33852)
0812 14:32:22.156207 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.156243 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.156704 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.156950 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.156952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.157180 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.157328 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.157331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.158298 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.158298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.158976 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.158978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.159447 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.159448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.159499 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.159950 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.159957 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.161466 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.162971 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.162977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.162979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.163162 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.163372 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.163373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.163373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.163455 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.163826 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.163829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.163880 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.163954 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.164191 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":93,"thread_start_us":70,"threads_started":1}
I20260812 14:32:22.164346 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.156089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.23.173.254:46821)
0812 14:32:22.156253 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.156257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.156314 (+ 57us) server_negotiation.cc:407] Connection header received
0812 14:32:22.156836 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.156838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.156857 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.156938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.157423 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.158221 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.159070 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.159379 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.159463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.161544 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.161552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.161552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.161559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.162896 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.163236 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.163239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.163239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.163314 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.163509 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.163510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.163510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.163728 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.163803 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.164002 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.164118 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260812 14:32:22.165035 29840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:22.165370 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.165265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36702)
0812 14:32:22.165304 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.165348 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:32:22.165500 29764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
I20260812 14:32:22.165865 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.165800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36704)
0812 14:32:22.165825 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.165848 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260812 14:32:22.165957 29840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:37975: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.173.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab.
Entry for principal kudu/127.23.173.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab.
I20260812 14:32:22.186268 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:37975
--webserver_interface=127.23.173.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab
--principal=kudu/127.23.173.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46821,127.23.173.253:37975,127.23.173.252:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:32:22.215196 29840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } attempt: 1
I20260812 14:32:22.215690 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.215553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36712)
0812 14:32:22.215616 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.215658 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:32:22.216105 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.216029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36722)
0812 14:32:22.216064 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.216088 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:32:22.216188 29840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:37975: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
W20260812 14:32:22.267194 29853 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:32:22.267383 29853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:22.267400 29853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:22.267418 29853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:22.268530 29853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:22.268566 29853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:22.268579 29853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:22.268597 29853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:22.268613 29853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:22.269994 29853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46821,127.23.173.253:37975,127.23.173.252:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab
--principal=kudu/127.23.173.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:22.270167 29853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:22.270339 29853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:22.275251 29865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:22.275364 29866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:22.275252 29869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:22.275380 29853 server_base.cc:1034] running on GCE node
I20260812 14:32:22.275820 29853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:22.276100 29853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:22.277241 29853 hybrid_clock.cc:648] HybridClock initialized: now 1786545142277227 us; error 31 us; skew 500 ppm
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:22.279662 29853 init.cc:377] Logged in from keytab as kudu/127.23.173.253@KRBTEST.COM (short username kudu)
I20260812 14:32:22.280117 29853 webserver.cc:460] Webserver started at http://127.23.173.253:37835/ using document root <none> and password file <none>
I20260812 14:32:22.280336 29853 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:22.280385 29853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:22.280524 29853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:22.281602 29853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data/instance:
uuid: "ed09e8221daa4ce990703367a605fc3e"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "e7efedd311d9c0a9b26db3e65bf7fd6b"
server_key_iv: "40b521ae322e8d5df453305ad5d60fe7"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.281957 29853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal/instance:
uuid: "ed09e8221daa4ce990703367a605fc3e"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "e7efedd311d9c0a9b26db3e65bf7fd6b"
server_key_iv: "40b521ae322e8d5df453305ad5d60fe7"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.283257 29853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:32:22.283973 29877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:22.284122 29853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:22.284206 29853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal
uuid: "ed09e8221daa4ce990703367a605fc3e"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "e7efedd311d9c0a9b26db3e65bf7fd6b"
server_key_iv: "40b521ae322e8d5df453305ad5d60fe7"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.284265 29853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:22.284333 29840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } attempt: 2
I20260812 14:32:22.284732 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.284622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36738)
0812 14:32:22.284671 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.284706 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:32:22.285063 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.284971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36748)
0812 14:32:22.285013 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.285039 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:32:22.285146 29840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:37975: Network error: Client connection negotiation failed: client connection to 127.23.173.253:37975: connect: Connection refused (error 111)
I20260812 14:32:22.295377 29853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:22.297147 29853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:22.297277 29853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:22.301047 29853 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.253:37975
I20260812 14:32:22.301079 29953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.253:37975 every 8 connection(s)
I20260812 14:32:22.301383 29853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
I20260812 14:32:22.302197 29954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:22.303627 29954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:22.311106 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29853
I20260812 14:32:22.311193 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-1/wal/instance
I20260812 14:32:22.311368 24247 external_mini_cluster.cc:1428] Setting key cdc5c7f93bf3ea83984799cc71ddd741
I20260812 14:32:22.311830 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.304287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.23.173.254:46821)
0812 14:32:22.304326 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.304328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.304541 (+ 213us) server_negotiation.cc:407] Connection header received
0812 14:32:22.305014 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.305016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.305031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.305085 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.305680 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.306153 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.307072 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.307268 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.307323 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.309215 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.309216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.309217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.309224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.310424 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.310781 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.310782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.310782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.310834 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.311115 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.311117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.311117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.311261 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.311322 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.311651 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.311768 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:32:22.311991 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.304250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46821 (local address 127.0.0.1:33860)
0812 14:32:22.304419 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.304460 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.304919 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.305165 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.305169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.305394 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.305594 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.305598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.306235 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.306236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.306993 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.306995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.307337 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.307338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.307457 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.307897 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.307906 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.309127 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.310501 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.310509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.310511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.310714 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.310890 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.310891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.310892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.311038 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.311326 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.311334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.311519 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.311633 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.311801 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260812 14:32:22.312486 29954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.252@KRBTEST.COM; defaulting to no policy
I20260812 14:32:22.319653 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.312731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36760)
0812 14:32:22.312776 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.312788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.312826 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.313041 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.313042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.313181 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.313237 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.313239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.314384 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.314385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.314880 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.314881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.315503 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.315503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.315552 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.315933 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.315939 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.317049 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.318623 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.318624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.318625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.318825 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.319015 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.319016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.319016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.319080 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.319338 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.319339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.319383 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.319450 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.319560 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:32:22.319774 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.312778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.23.173.253:37975)
0812 14:32:22.312910 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.312915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.312932 (+ 17us) server_negotiation.cc:407] Connection header received
0812 14:32:22.312953 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.312955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.312965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.313030 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.313304 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.314302 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.314959 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.315423 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.315502 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.317152 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.317159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.317159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.317165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.318539 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.318902 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.318904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.318905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.318958 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.319151 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.319151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.319152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.319276 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.319336 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.319480 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.319637 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260812 14:32:22.320325 29954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:22.320642 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.320535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57872)
0812 14:32:22.320578 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.320621 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 14:32:22.320827 29882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
I20260812 14:32:22.321241 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.321158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57882)
0812 14:32:22.321197 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.321223 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:32:22.321329 29954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:45125: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab.
Entry for principal kudu/127.23.173.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab.
I20260812 14:32:22.335786 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:45125
--webserver_interface=127.23.173.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab
--principal=kudu/127.23.173.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:46821,127.23.173.253:37975,127.23.173.252:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:32:22.370577 29954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } attempt: 1
I20260812 14:32:22.371119 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.370980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57898)
0812 14:32:22.371045 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.371087 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 14:32:22.371510 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.371401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57910)
0812 14:32:22.371464 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.371491 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 14:32:22.371606 29954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:45125: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
I20260812 14:32:22.376278 29840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } attempt: 3
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:32:22.383054 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.376587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36764)
0812 14:32:22.376642 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.376653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.376713 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.376840 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.376840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.376983 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.377051 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.377053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.377935 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.377936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.378462 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.378464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.378949 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.378950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.379006 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.379455 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.379460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.380646 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.382217 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.382217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.382218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.382372 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.382554 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.382555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.382555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.382598 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.382832 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.382833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.382870 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.382914 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.382990 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:32:22.383107 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.376636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.23.173.253:37975)
0812 14:32:22.376700 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.376703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.376715 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:32:22.376760 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.376761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.376774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.376834 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.377111 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.377863 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.378546 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.378876 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.378945 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.380725 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.380726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.380727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.380734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.382141 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.382432 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.382433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.382434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.382490 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.382649 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.382650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.382650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.382776 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.382833 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.382953 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.383028 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:32:22.383771 29840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:22.384099 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.384011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57920)
0812 14:32:22.384051 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.384080 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:32:22.384459 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.384358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57930)
0812 14:32:22.384396 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.384420 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:32:22.384537 29840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:45125: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
I20260812 14:32:22.415670 29840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } attempt: 1
I20260812 14:32:22.416113 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.416005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57934)
0812 14:32:22.416056 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.416088 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:32:22.416445 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.416354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57950)
0812 14:32:22.416394 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.416419 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:32:22.416546 29840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:45125: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
W20260812 14:32:22.418095 29963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:32:22.418274 29963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:22.418289 29963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:22.418313 29963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:22.419404 29963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:22.419436 29963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:22.419449 29963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:22.419461 29963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:22.419471 29963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:22.420917 29963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:46821,127.23.173.253:37975,127.23.173.252:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab
--principal=kudu/127.23.173.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:22.421118 29963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:22.421288 29963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:32:22.425882 29963 server_base.cc:1034] running on GCE node
W20260812 14:32:22.425859 29973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:22.425895 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:22.425865 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:22.426294 29963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:22.426581 29963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:22.427736 29963 hybrid_clock.cc:648] HybridClock initialized: now 1786545142427715 us; error 31 us; skew 500 ppm
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:22.430078 29963 init.cc:377] Logged in from keytab as kudu/127.23.173.252@KRBTEST.COM (short username kudu)
I20260812 14:32:22.430510 29963 webserver.cc:460] Webserver started at http://127.23.173.252:45785/ using document root <none> and password file <none>
I20260812 14:32:22.430730 29963 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:22.430779 29963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:22.430902 29963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:22.431957 29963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data/instance:
uuid: "754fccc169d84e0b9b520b63ce5b9f3a"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "57d3ca686567f1d1f4a7996f9e07eab1"
server_key_iv: "b9e9999144e50e784756885d810b8db9"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.432296 29963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal/instance:
uuid: "754fccc169d84e0b9b520b63ce5b9f3a"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "57d3ca686567f1d1f4a7996f9e07eab1"
server_key_iv: "b9e9999144e50e784756885d810b8db9"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.433498 29963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:32:22.434139 29987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:22.434283 29963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:22.434355 29963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal
uuid: "754fccc169d84e0b9b520b63ce5b9f3a"
format_stamp: "Formatted at 2026-08-12 14:32:22 on dist-test-slave-1l3l"
server_key: "57d3ca686567f1d1f4a7996f9e07eab1"
server_key_iv: "b9e9999144e50e784756885d810b8db9"
server_key_version: "encryptionkey@0"
I20260812 14:32:22.434417 29963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:22.439762 29954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } attempt: 2
I20260812 14:32:22.440119 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.439997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57964)
0812 14:32:22.440049 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.440089 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:32:22.440374 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.440309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57972)
0812 14:32:22.440338 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.440359 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 14:32:22.440474 29954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:45125: Network error: Client connection negotiation failed: client connection to 127.23.173.252:45125: connect: Connection refused (error 111)
I20260812 14:32:22.444037 29963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:22.445597 29963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:22.445703 29963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:22.449487 29963 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.252:45125
I20260812 14:32:22.449517 30090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.252:45125 every 8 connection(s)
I20260812 14:32:22.449831 29963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
I20260812 14:32:22.450085 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 29963
I20260812 14:32:22.450268 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544840008271-24247-0/minicluster-data/master-2/wal/instance
I20260812 14:32:22.450449 24247 external_mini_cluster.cc:1428] Setting key 7df9e0424f4ddbfbde8db345b42dc09b
I20260812 14:32:22.450722 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:22.453023 30091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:22.460536 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.453633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.23.173.254:46821)
0812 14:32:22.453680 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.453682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.453835 (+ 153us) server_negotiation.cc:407] Connection header received
0812 14:32:22.453897 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.453897 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.453907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.453960 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.454353 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.454938 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.455892 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.456185 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.456260 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.458220 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.458221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.458221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.458228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.459447 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.459787 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.459788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.459789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.459838 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.460036 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.460036 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.460037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.460135 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.460184 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.460359 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.460438 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:32:22.460721 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.453579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:46821 (local address 127.0.0.1:33872)
0812 14:32:22.453742 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.453774 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.453836 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.453994 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.453997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.454219 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.454283 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.454285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.455011 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.455012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.455815 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.455818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.456263 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.456264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.456315 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.456700 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.456707 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.458138 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.459527 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.459533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.459535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.459729 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.459892 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.459892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.459893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.459978 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.460189 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.460193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.460250 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.460338 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.460503 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260812 14:32:22.460729 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.450888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.23.173.252:45125)
0812 14:32:22.451091 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.451100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.451129 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:32:22.451808 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.451815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.451863 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.451973 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.452772 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.453876 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.454764 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.455250 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.455367 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.457828 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.457836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.457837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.457844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.459409 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.459762 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.459765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.459765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.459848 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.460018 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.460019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.460019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.460230 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.460387 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.460435 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.460560 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260812 14:32:22.461298 30091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:22.467391 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.461509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:37975 (local address 127.0.0.1:36776)
0812 14:32:22.461563 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.461573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.461612 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.461725 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.461726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.461832 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.461868 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.461869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.462536 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.462537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.463057 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.463059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.463451 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.463452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.463512 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.463889 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.463893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.465067 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.466514 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.466515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.466515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.466674 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.466874 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.466874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.466875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.466915 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.467140 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.467140 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.467170 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.467220 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.467305 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 14:32:22.467430 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.461566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.23.173.253:37975)
0812 14:32:22.461619 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.461621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.461631 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:32:22.461657 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.461658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.461666 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.461722 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.461932 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.462464 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.463129 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.463392 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.463441 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.465151 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.465152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.465152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.465161 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.466452 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.466744 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.466746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.466746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.466811 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.466955 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.466955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.466956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.467085 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.467150 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.467276 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.467352 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:32:22.467737 30091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:22.468250 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.461461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.23.173.254:46821)
0812 14:32:22.461495 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.461497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.461506 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:32:22.461574 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.461575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.461586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.461628 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.462153 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.462630 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.463273 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.463454 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.463511 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.465697 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.465698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.465698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.465704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.467162 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.467444 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.467445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.467445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.467500 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.467787 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.467788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.467788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.467913 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.468102 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.468135 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.468192 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:32:22.474483 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.467961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57988)
0812 14:32:22.468256 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.468266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.468310 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.468433 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.468434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.468569 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.468601 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.468603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.469272 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.469273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.469996 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.469998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.470472 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.470473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.470511 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.470864 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.470868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.472195 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.473568 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.473569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.473569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.473750 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.473931 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.473932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.473932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.473992 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.474187 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.474188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.474232 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.474304 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.474394 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260812 14:32:22.474630 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.467990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.23.173.252:45125)
0812 14:32:22.468032 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.468034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.468352 (+ 318us) server_negotiation.cc:407] Connection header received
0812 14:32:22.468367 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.468367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.468374 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.468406 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.468658 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.469209 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.470099 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.470406 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.470471 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.472279 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.472281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.472281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.472287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.473483 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.473820 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.473821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.473822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.473876 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.474047 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.474048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.474048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.474135 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.474178 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.474325 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.474439 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":24}
I20260812 14:32:22.475476 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.468842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.23.173.253:37975)
0812 14:32:22.468874 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.468876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.468884 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:32:22.468902 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.468902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.468911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.468969 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.469353 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.470028 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.470799 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.471010 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.471071 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.473047 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.473048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.473048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.473055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.474436 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.474809 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.474810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.474810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.474863 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.475052 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.475053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.475054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.475158 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.475288 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.475366 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.475422 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:32:22.476236 30091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: Bootstrap starting.
I20260812 14:32:22.476877 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:22.477262 30091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:22.477571 30091 log.cc:824] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:22.478272 30091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: No bootstrap required, opened a new log
I20260812 14:32:22.478016 29801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f06bf2f88884c5283e1befea4e5ac6e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33880
I20260812 14:32:22.479641 30091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.479810 30091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fccc169d84e0b9b520b63ce5b9f3a, State: Initialized, Role: FOLLOWER
I20260812 14:32:22.479894 30091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.480204 30091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:22.480189 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.480290 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:32:22.481801 30118 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:22.481851 30118 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:22.482092 29963 master_runner.cc:186] Error getting master registration for 127.23.173.254:46821: OK, instance_id { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" instance_seqno: 1786545142151518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:32:22.484814 29963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:32:22.491696 29840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } attempt: 2
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:32:22.499150 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.492012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:57992)
0812 14:32:22.492064 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.492074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.492117 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.492263 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.492264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.492398 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.492439 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.492440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.493291 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.493292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.494102 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.494104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.494551 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.494552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.494604 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.495065 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.495070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.496386 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.498079 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.498080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.498081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.498304 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.498495 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.498496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.498496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.498563 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.498850 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.498851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.498897 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.498957 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.499049 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:32:22.499158 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.492053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.23.173.252:45125)
0812 14:32:22.492094 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.492096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.492107 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:32:22.492160 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.492161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.492187 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.492254 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.492496 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.493217 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.494175 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.494481 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.494550 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.496466 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.496467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.496468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.496475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.498001 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.498371 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.498372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.498372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.498438 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.498615 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.498616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.498616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.498789 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.498858 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.498988 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.499067 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:32:22.500489 29840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e: Bootstrap starting.
I20260812 14:32:22.501116 29840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:22.501377 29840 log.cc:824] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:22.502167 29840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e: No bootstrap required, opened a new log
I20260812 14:32:22.503176 29840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.503289 29840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f06bf2f88884c5283e1befea4e5ac6e, State: Initialized, Role: FOLLOWER
I20260812 14:32:22.503362 29840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.503629 30124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.503693 29840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:22.503731 30124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:22.503837 29801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 14:32:22.503950 29801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:22.504773 29801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.505388 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "754fccc169d84e0b9b520b63ce5b9f3a"
I20260812 14:32:22.505514 30044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [term 0 FOLLOWER]: Advancing to term 1
W20260812 14:32:22.505594 30139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:22.505640 30139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:22.506366 30044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f06bf2f88884c5283e1befea4e5ac6e in term 1.
I20260812 14:32:22.507884 29801 leader_election.cc:288] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [CANDIDATE]: Term 1 election: Requested vote from peers ed09e8221daa4ce990703367a605fc3e (127.23.173.253:37975), 754fccc169d84e0b9b520b63ce5b9f3a (127.23.173.252:45125)
I20260812 14:32:22.507274 29731 master_runner.cc:186] Error getting master registration for 127.23.173.253:37975: OK, instance_id { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" instance_seqno: 1786545142300012 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:32:22.509574 29731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:32:22.509728 29762 leader_election.cc:302] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f06bf2f88884c5283e1befea4e5ac6e, 754fccc169d84e0b9b520b63ce5b9f3a; no voters:
I20260812 14:32:22.509899 30124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:22.510051 30124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [term 1 LEADER]: Becoming Leader. State: Replica: 4f06bf2f88884c5283e1befea4e5ac6e, State: Running, Role: LEADER
I20260812 14:32:22.510138 30124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.511791 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f06bf2f88884c5283e1befea4e5ac6e. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.511898 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:22.512152 30154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:22.512504 30154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:22.524364 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.512427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.23.173.254:46821)
0812 14:32:22.512474 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.512476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.514277 (+ 1801us) server_negotiation.cc:407] Connection header received
0812 14:32:22.514363 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.514364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.514373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.514398 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.516973 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.517673 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.519355 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.519657 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.519693 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.521173 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.521174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.521175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.521181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.523131 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.523520 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.523521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.523521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.523575 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.523802 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.523803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.523803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.523924 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.524001 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.524203 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.524276 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:32:22.524389 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.512237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.23.173.252:45125)
0812 14:32:22.512274 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.512276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.513301 (+ 1025us) server_negotiation.cc:407] Connection header received
0812 14:32:22.513392 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.513393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.513402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.513442 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.515142 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.515929 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.518697 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.518993 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.519032 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.521275 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.521277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.521277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.521284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.522774 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.523323 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.523324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.523325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.523381 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.523683 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.523684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.523684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.523835 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.523909 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.524222 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.524301 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:32:22.526906 30044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [term 1 FOLLOWER]: Refusing update from remote peer 4f06bf2f88884c5283e1befea4e5ac6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:32:22.527227 30143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:32:22.528522 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f06bf2f88884c5283e1befea4e5ac6e. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.528616 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:22.531589 29954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } attempt: 3
I20260812 14:32:22.533491 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.513039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.23.173.253:37975)
0812 14:32:22.513084 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.513086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.519836 (+ 6750us) server_negotiation.cc:407] Connection header received
0812 14:32:22.519940 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.519942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.519953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.519982 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.524723 (+ 4741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.525324 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.525975 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.526243 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.526311 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.530250 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.530263 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.530264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.530271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.531914 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.532492 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.532494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.532494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.532563 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.532820 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.532821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.532821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.532944 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.533018 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.533322 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.533401 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:32:22.535669 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.535770 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: This master's current role is: LEADER
Aug 12 14:32:22 dist-test-slave-1l3l krb5kdc[29709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:32:22.536724 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.536818 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:22.537619 30161 mvcc.cc:205] Tried to move back new op lower bound from 7317688903787384832 to 7317688903722225664. Current Snapshot: MvccSnapshot[applied={T|T < 7317688903787384832}]
I20260812 14:32:22.540041 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.531871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:45125 (local address 127.0.0.1:58016)
0812 14:32:22.531974 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:22.531984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:22.532021 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:22.532228 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:22.532229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:22.532384 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:22.532423 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.532424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.533285 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.533286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.534060 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:22.534063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.534780 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.534781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:22.534851 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.535316 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:22.535332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:22.537177 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:22.538987 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.538988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.538989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.539187 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.539388 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:22.539389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:22.539390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:22.539452 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:22.539749 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:22.539750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:22.539798 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 14:32:22.539870 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:32:22.539958 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 14:32:22.540052 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:22.532089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.23.173.252:45125)
0812 14:32:22.532121 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.532123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.532129 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:32:22.532146 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.532146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.532154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.532194 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.532496 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.533196 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.534344 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:22.534660 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:22.534719 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:22.537338 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:22.537340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:22.537340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:22.537347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:22.538899 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.539274 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.539275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.539275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.539335 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:22.539525 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:22.539526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:22.539537 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:22.539662 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:22.539752 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:22.539896 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:32:22.539969 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:32:22.540117 30154 catalog_manager.cc:1347] Generated new cluster ID: b43364f4572744ea951cb8f5c4ab4d92
I20260812 14:32:22.540154 30154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:22.541536 29954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: Bootstrap starting.
I20260812 14:32:22.542183 29954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:22.542455 29954 log.cc:824] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:22.543272 29954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: No bootstrap required, opened a new log
I20260812 14:32:22.545336 29954 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.545554 29954 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed09e8221daa4ce990703367a605fc3e, State: Initialized, Role: FOLLOWER
I20260812 14:32:22.545650 29954 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } }
I20260812 14:32:22.545964 29954 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:22.545989 29921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed09e8221daa4ce990703367a605fc3e"
I20260812 14:32:22.546085 29921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:22.546830 29921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f06bf2f88884c5283e1befea4e5ac6e in term 1.
I20260812 14:32:22.546998 29919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [term 1 FOLLOWER]: Refusing update from remote peer 4f06bf2f88884c5283e1befea4e5ac6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:32:22.547370 30124 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:32:22.547808 29887 tablet.cc:2369] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:32:22.548051 30186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:22.548105 30186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:22.548271 30166 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.548384 30166 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:22.549754 30191 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.549853 30191 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:22.550083 30166 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f06bf2f88884c5283e1befea4e5ac6e. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.550139 30166 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:22.563757 30124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f06bf2f88884c5283e1befea4e5ac6e" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 46821 } } peers { permanent_uuid: "ed09e8221daa4ce990703367a605fc3e" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 37975 } } peers { permanent_uuid: "754fccc169d84e0b9b520b63ce5b9f3a" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 45125 } } }
I20260812 14:32:22.563844 30124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:22.566304 30154 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:22.566967 30154 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:22.579613 30154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f06bf2f88884c5283e1befea4e5ac6e: Generated new TSK 0
I20260812 14:32:22.579798 30154 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:32:22.611811 29848 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.610025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.23.173.254:46821)
0812 14:32:22.610071 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.610073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.611062 (+ 989us) server_negotiation.cc:407] Connection header received
0812 14:32:22.611170 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.611171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.611182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.611212 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.611696 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33900: BlockingRecv error: recv got EOF from 127.0.0.1:33900 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:32:22.612994 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.610414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.23.173.253:37975)
0812 14:32:22.610466 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.610468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.612274 (+ 1806us) server_negotiation.cc:407] Connection header received
0812 14:32:22.612522 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.612523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.612535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.612562 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.612934 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36804: BlockingRecv error: recv got EOF from 127.0.0.1:36804 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 14:32:22.613766 30095 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.610767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.23.173.252:45125)
0812 14:32:22.610808 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.610810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.612473 (+ 1663us) server_negotiation.cc:407] Connection header received
0812 14:32:22.613358 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.613359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.613372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.613400 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.613706 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58022: BlockingRecv error: recv got EOF from 127.0.0.1:58022 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:32:22.616290 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.615560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.23.173.253:37975)
0812 14:32:22.615593 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.615595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.615817 (+ 222us) server_negotiation.cc:407] Connection header received
0812 14:32:22.615913 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.615914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.615929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.615955 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.616251 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36818: BlockingRecv error: recv got EOF from 127.0.0.1:36818 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:32:22.619390 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.618661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.23.173.253:37975)
0812 14:32:22.618705 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.618706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.618928 (+ 222us) server_negotiation.cc:407] Connection header received
0812 14:32:22.619016 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.619017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.619026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.619054 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.619345 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36828: BlockingRecv error: recv got EOF from 127.0.0.1:36828 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:32:22.623260 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.622536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.23.173.253:37975)
0812 14:32:22.622566 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.622568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.622794 (+ 226us) server_negotiation.cc:407] Connection header received
0812 14:32:22.622879 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.622880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.622890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.622916 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.623216 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36840: BlockingRecv error: recv got EOF from 127.0.0.1:36840 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 14:32:22.631057 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.630287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.23.173.253:37975)
0812 14:32:22.630322 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.630323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.630575 (+ 252us) server_negotiation.cc:407] Connection header received
0812 14:32:22.630661 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.630662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.630671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.630696 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.631017 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36842: BlockingRecv error: recv got EOF from 127.0.0.1:36842 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:32:22.641068 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.640240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.23.173.253:37975)
0812 14:32:22.640274 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.640276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.640556 (+ 280us) server_negotiation.cc:407] Connection header received
0812 14:32:22.640653 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.640654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.640664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.640691 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.641024 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36852: BlockingRecv error: recv got EOF from 127.0.0.1:36852 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:32:22.658986 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.658314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.23.173.253:37975)
0812 14:32:22.658357 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.658360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.658532 (+ 172us) server_negotiation.cc:407] Connection header received
0812 14:32:22.658583 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.658584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.658594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.658641 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.658938 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36856: BlockingRecv error: recv got EOF from 127.0.0.1:36856 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:32:22.692734 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.692076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.23.173.253:37975)
0812 14:32:22.692116 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.692118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.692253 (+ 135us) server_negotiation.cc:407] Connection header received
0812 14:32:22.692302 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.692303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.692314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.692362 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.692680 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36872: BlockingRecv error: recv got EOF from 127.0.0.1:36872 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:32:22.758395 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.757717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.23.173.253:37975)
0812 14:32:22.757779 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.757781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.757874 (+ 93us) server_negotiation.cc:407] Connection header received
0812 14:32:22.757925 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.757926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.757935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.758001 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.758335 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36876: BlockingRecv error: recv got EOF from 127.0.0.1:36876 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 14:32:22.888230 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:22.887529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.23.173.253:37975)
0812 14:32:22.887582 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:22.887584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:22.887687 (+ 103us) server_negotiation.cc:407] Connection header received
0812 14:32:22.887791 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:22.887791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:22.887802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:22.887863 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:22.888166 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36890: BlockingRecv error: recv got EOF from 127.0.0.1:36890 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260812 14:32:23.145924 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:23.145323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.23.173.253:37975)
0812 14:32:23.145387 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:23.145389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:23.145399 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:32:23.145486 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:23.145487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:23.145498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:23.145545 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:23.145862 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36906: BlockingRecv error: recv got EOF from 127.0.0.1:36906 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:32:23.482261 30118 catalog_manager.cc:1259] Loaded cluster ID: b43364f4572744ea951cb8f5c4ab4d92
I20260812 14:32:23.482335 30118 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: loading cluster ID for follower catalog manager: success
I20260812 14:32:23.483142 30118 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: acquiring CA information for follower catalog manager: success
I20260812 14:32:23.483446 30118 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 754fccc169d84e0b9b520b63ce5b9f3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:32:23.548522 30186 catalog_manager.cc:1259] Loaded cluster ID: b43364f4572744ea951cb8f5c4ab4d92
I20260812 14:32:23.548614 30186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: loading cluster ID for follower catalog manager: success
I20260812 14:32:23.549435 30186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: acquiring CA information for follower catalog manager: success
I20260812 14:32:23.549746 30186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed09e8221daa4ce990703367a605fc3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:32:23.661871 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:23.659131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.23.173.253:37975)
0812 14:32:23.659246 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:23.659248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:23.659258 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:32:23.659285 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:23.659285 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:23.659338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:23.659417 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:23.659542 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.660086 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.660854 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.661137 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.661207 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:23.661410 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:23.661590 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:23.661674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:23.661735 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:32:23.661774 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260812 14:32:23.665910 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:23.663732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.23.173.252:45125)
0812 14:32:23.663842 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:23.663844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:23.663854 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:32:23.663893 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:23.663893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:23.663928 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:23.664002 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:23.664086 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.664614 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.665296 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.665534 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.665588 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:23.665685 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:23.665771 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:23.665817 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:23.665831 (+ 14us) server_negotiation.cc:299] Negotiation successful
0812 14:32:23.665848 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260812 14:32:23.669986 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:23.667485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.23.173.252:45125)
0812 14:32:23.667530 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:23.667532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:23.667769 (+ 237us) server_negotiation.cc:407] Connection header received
0812 14:32:23.667832 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:23.667833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:23.667851 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:23.667895 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:23.667980 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.668515 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.669246 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.669541 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.669600 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:23.669765 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:23.669797 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:23.669846 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:23.669895 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 14:32:23.669917 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:32:23.670387 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:23.667686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.23.173.253:37975)
0812 14:32:23.667759 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:23.667760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:23.667898 (+ 138us) server_negotiation.cc:407] Connection header received
0812 14:32:23.667941 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:23.667942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:23.667954 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:23.668000 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:23.668182 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.668894 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.669656 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.669913 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.669977 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:23.670158 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:23.670190 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:23.670214 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:23.670299 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:32:23.670321 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:32:23.670884 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:23.667378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.23.173.254:46821)
0812 14:32:23.667555 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:23.667557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:23.667963 (+ 406us) server_negotiation.cc:407] Connection header received
0812 14:32:23.668013 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:23.668014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:23.668066 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:23.668152 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:23.668281 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.669013 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.670010 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:23.670282 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:23.670341 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:23.670575 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:23.670675 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:23.670733 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:23.670771 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:32:23.670795 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":53,"threads_started":1}
I20260812 14:32:23.672387 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29731
I20260812 14:32:23.684706 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29853
I20260812 14:32:23.692826 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 29963
2026-08-12T14:32:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2939 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:32:23 dist-test-slave-1l3l krb5kdc[30339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:32:23 dist-test-slave-1l3l krb5kdc[30339](info): set up 2 sockets
Aug 12 14:32:23 dist-test-slave-1l3l krb5kdc[30339](info): commencing operation
krb5kdc: starting...
W20260812 14:32:24.867604 24247 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/test-user.keytab.
Aug 12 14:32:24 dist-test-slave-1l3l krb5kdc[30339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
Entry for principal kudu/127.23.173.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab.
I20260812 14:32:24.985891 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37421
--webserver_interface=127.23.173.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37421,127.23.173.253:42351,127.23.173.252:37441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:32:25.070418 30359 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:32:25.070637 30359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:25.070662 30359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:25.070688 30359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:25.071863 30359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:25.071903 30359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:25.071921 30359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:25.071938 30359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:25.071954 30359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:25.073432 30359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37421,127.23.173.253:42351,127.23.173.252:37441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.254.keytab
--principal=kudu/127.23.173.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.254:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:25.073688 30359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:25.073907 30359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:25.078644 30379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:25.078682 30375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:25.078738 30359 server_base.cc:1034] running on GCE node
W20260812 14:32:25.078677 30377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:25.079131 30359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:25.079401 30359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:25.080546 30359 hybrid_clock.cc:648] HybridClock initialized: now 1786545145080531 us; error 31 us; skew 500 ppm
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:25.086371 30359 init.cc:377] Logged in from keytab as kudu/127.23.173.254@KRBTEST.COM (short username kudu)
I20260812 14:32:25.086826 30359 webserver.cc:460] Webserver started at http://127.23.173.254:37573/ using document root <none> and password file <none>
I20260812 14:32:25.087013 30359 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:25.087051 30359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:25.087152 30359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:25.088217 30359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data/instance:
uuid: "737ef3931b564de485023564c3b5015a"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "a360df900dfec8739ca117953655eff7"
server_key_iv: "4472bf5a7365168ad7ff08cda453b913"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.088553 30359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal/instance:
uuid: "737ef3931b564de485023564c3b5015a"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "a360df900dfec8739ca117953655eff7"
server_key_iv: "4472bf5a7365168ad7ff08cda453b913"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.089789 30359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:32:25.090490 30387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:25.090668 30359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:25.090752 30359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal
uuid: "737ef3931b564de485023564c3b5015a"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "a360df900dfec8739ca117953655eff7"
server_key_iv: "4472bf5a7365168ad7ff08cda453b913"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.090811 30359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:25.132618 30359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:25.134938 30359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:25.135066 30359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:25.139333 30359 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.254:37421
I20260812 14:32:25.139397 30479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.254:37421 every 8 connection(s)
I20260812 14:32:25.139760 30359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/data/info.pb
I20260812 14:32:25.140687 30480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:25.141393 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30359
I20260812 14:32:25.141510 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-0/wal/instance
I20260812 14:32:25.141696 24247 external_mini_cluster.cc:1428] Setting key 894af5ba27d4e259b68b3dbf1c7fc5dd
I20260812 14:32:25.142354 30480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.253@KRBTEST.COM; defaulting to no policy
I20260812 14:32:25.151945 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.143114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37421 (local address 127.0.0.1:39668)
0812 14:32:25.143289 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.143325 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.143978 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.144303 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.144306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.144668 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.144867 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.144874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.145874 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.145875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.146547 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.146549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.146995 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.146997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.147070 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.147561 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.147569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.148965 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.150534 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.150548 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.150550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.150760 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.150999 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.151000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.151000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.151091 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.151407 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.151411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.151469 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.151556 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.151785 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":68,"thread_start_us":88,"threads_started":1}
I20260812 14:32:25.151938 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.143321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.23.173.254:37421)
0812 14:32:25.143451 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.143456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.143495 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:32:25.144169 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.144172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.144190 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.144274 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.144971 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.145784 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.146622 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.146921 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.147015 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.149050 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.149058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.149058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.149064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.150456 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.150843 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.150847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.150847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.150936 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.151142 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.151143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.151143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.151352 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.151413 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.151580 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.151678 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.253@KRBTEST.COM" created.
I20260812 14:32:25.152733 30480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:25.153056 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.152950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53636)
0812 14:32:25.152991 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.153034 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:32:25.153203 30390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
I20260812 14:32:25.153612 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.153522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53650)
0812 14:32:25.153571 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.153596 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 14:32:25.153746 30480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:42351: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab.
Entry for principal kudu/127.23.173.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab.
I20260812 14:32:25.171114 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:42351
--webserver_interface=127.23.173.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab
--principal=kudu/127.23.173.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37421,127.23.173.253:42351,127.23.173.252:37441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:32:25.202984 30480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } attempt: 1
I20260812 14:32:25.203550 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.203395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53656)
0812 14:32:25.203474 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.203516 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 14:32:25.203953 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.203858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53664)
0812 14:32:25.203904 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.203931 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 14:32:25.204061 30480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:42351: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
W20260812 14:32:25.255191 30489 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:32:25.255453 30489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:25.255481 30489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:25.255507 30489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:25.256783 30489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:25.256830 30489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:25.256848 30489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:25.256866 30489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:25.256880 30489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:25.258388 30489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37421,127.23.173.253:42351,127.23.173.252:37441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.253.keytab
--principal=kudu/127.23.173.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.253:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:25.258639 30489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:25.258858 30489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:25.263684 30497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:25.263684 30496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:25.263684 30499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:25.263989 30489 server_base.cc:1034] running on GCE node
I20260812 14:32:25.264207 30489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:25.264467 30489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:25.265614 30489 hybrid_clock.cc:648] HybridClock initialized: now 1786545145265600 us; error 31 us; skew 500 ppm
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:25.268177 30489 init.cc:377] Logged in from keytab as kudu/127.23.173.253@KRBTEST.COM (short username kudu)
I20260812 14:32:25.268630 30489 webserver.cc:460] Webserver started at http://127.23.173.253:37817/ using document root <none> and password file <none>
I20260812 14:32:25.268833 30489 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:25.268873 30489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:25.268976 30489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:25.270076 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data/instance:
uuid: "7c85c2ca9f5b48b897723d4ff26f4446"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "8706087bdf504f17e97d00ff000bc733"
server_key_iv: "100020cfe44e7b6fe9cec43699c6a6ad"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.270373 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal/instance:
uuid: "7c85c2ca9f5b48b897723d4ff26f4446"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "8706087bdf504f17e97d00ff000bc733"
server_key_iv: "100020cfe44e7b6fe9cec43699c6a6ad"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.271637 30489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:32:25.272219 30480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } attempt: 2
I20260812 14:32:25.272450 30509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:25.272632 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.272519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53668)
0812 14:32:25.272565 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.272603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 14:32:25.272836 30489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:32:25.272938 30489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal
uuid: "7c85c2ca9f5b48b897723d4ff26f4446"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "8706087bdf504f17e97d00ff000bc733"
server_key_iv: "100020cfe44e7b6fe9cec43699c6a6ad"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.272986 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.272843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53680)
0812 14:32:25.272917 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.272954 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 14:32:25.273015 30489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 14:32:25.273069 30480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.253:42351: Network error: Client connection negotiation failed: client connection to 127.23.173.253:42351: connect: Connection refused (error 111)
I20260812 14:32:25.282474 30489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:25.284629 30489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:25.284767 30489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:25.288652 30489 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.253:42351
I20260812 14:32:25.288697 30590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.253:42351 every 8 connection(s)
I20260812 14:32:25.289038 30489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/data/info.pb
I20260812 14:32:25.289912 30591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:25.291438 30591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:25.297027 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30489
I20260812 14:32:25.297137 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-1/wal/instance
I20260812 14:32:25.297377 24247 external_mini_cluster.cc:1428] Setting key ad2c2251f57a653dc3572ad52a21ed19
I20260812 14:32:25.300151 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.292161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.23.173.254:37421)
0812 14:32:25.292211 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.292213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.292416 (+ 203us) server_negotiation.cc:407] Connection header received
0812 14:32:25.292884 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.292885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.292896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.292958 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.293607 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.294097 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.295051 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.295256 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.295302 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.297475 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.297477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.297477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.297485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.298782 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.299230 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.299231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.299231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.299297 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.299499 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.299499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.299500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.299627 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.299690 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.299970 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.300061 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:32:25.300226 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.292119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37421 (local address 127.0.0.1:39672)
0812 14:32:25.292291 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.292339 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.292801 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.293021 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.293024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.293290 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.293527 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.293532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.294174 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.294174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.294977 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.294980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.295332 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.295333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.295478 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.295948 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.295957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.297373 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.298877 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.298886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.298889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.299163 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.299356 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.299357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.299357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.299450 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.299682 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.299685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.299839 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.299934 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.300056 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260812 14:32:25.300745 30591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.252@KRBTEST.COM; defaulting to no policy
I20260812 14:32:25.307955 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.300969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53694)
0812 14:32:25.301011 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.301021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.301059 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.301297 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.301298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.301456 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.301525 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.301528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.302382 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.302383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.303141 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.303144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.303631 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.303631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.303726 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.304075 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.304079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.305380 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.307001 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.307002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.307003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.307152 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.307328 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.307329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.307329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.307389 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.307639 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.307640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.307678 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.307755 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.307857 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 14:32:25.308074 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.301044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.23.173.253:42351)
0812 14:32:25.301175 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.301179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.301193 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:32:25.301208 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.301210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.301220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.301272 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.301598 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.302296 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.303238 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.303555 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.303617 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.305470 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.305478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.305478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.305485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.306924 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.307222 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.307225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.307226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.307273 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.307439 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.307440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.307440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.307578 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.307649 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.307784 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.307937 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.252@KRBTEST.COM" created.
I20260812 14:32:25.308728 30591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:25.309065 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.308946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35930)
0812 14:32:25.308989 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.309038 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:32:25.309193 30513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
I20260812 14:32:25.309597 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.309508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35938)
0812 14:32:25.309545 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.309570 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:32:25.309695 30591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:37441: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab.
Entry for principal kudu/127.23.173.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab.
I20260812 14:32:25.325153 24247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhwVqn/build/release/bin/kudu
/tmp/dist-test-taskOhwVqn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:37441
--webserver_interface=127.23.173.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab
--principal=kudu/127.23.173.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.254:37421,127.23.173.253:42351,127.23.173.252:37441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:32:25.358914 30591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } attempt: 1
I20260812 14:32:25.359408 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.359285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35942)
0812 14:32:25.359340 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.359375 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:32:25.359817 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.359667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35958)
0812 14:32:25.359741 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.359785 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260812 14:32:25.359939 30591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:37441: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
I20260812 14:32:25.364184 30480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } attempt: 3
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:32:25.371163 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.364547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.23.173.253:42351)
0812 14:32:25.364598 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.364600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.364642 (+ 42us) server_negotiation.cc:407] Connection header received
0812 14:32:25.364673 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.364674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.364687 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.364738 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.365044 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.365822 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.366486 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.366831 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.366902 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.368831 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.368832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.368833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.368840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.370153 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.370523 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.370523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.370524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.370575 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.370749 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.370749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.370749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.370848 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.370891 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.371040 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.371101 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:32:25.371196 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.364455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53708)
0812 14:32:25.364515 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.364527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.364580 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.364754 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.364755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.364910 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.364980 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.364982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.365895 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.365896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.366415 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.366416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.366895 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.366895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.366947 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.367418 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.367422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.368721 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.370234 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.370235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.370236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.370458 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.370627 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.370628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.370628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.370695 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.370894 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.370895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.370945 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.371005 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.371105 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 14:32:25.371824 30480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:32:25.372165 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.372051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35970)
0812 14:32:25.372101 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.372139 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:32:25.372473 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.372393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35978)
0812 14:32:25.372433 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.372457 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:32:25.372553 30480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:37441: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
I20260812 14:32:25.403731 30480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } attempt: 1
I20260812 14:32:25.404217 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.404072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35980)
0812 14:32:25.404137 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.404182 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:32:25.404608 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.404511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35990)
0812 14:32:25.404558 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.404586 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:32:25.404706 30480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:37441: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
W20260812 14:32:25.410032 30599 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:32:25.410224 30599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:32:25.410243 30599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:32:25.410279 30599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:32:25.411401 30599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:32:25.411441 30599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:32:25.411454 30599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:32:25.411466 30599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:32:25.411476 30599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:32:25.412917 30599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.254:37421,127.23.173.253:42351,127.23.173.252:37441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/krb5kdc/kudu_127.23.173.252.keytab
--principal=kudu/127.23.173.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.252:37441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:32:25.413121 30599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:32:25.413290 30599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:32:25.417861 30616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:25.417852 30609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:32:25.417852 30611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:32:25.418013 30599 server_base.cc:1034] running on GCE node
I20260812 14:32:25.418365 30599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:32:25.418635 30599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:32:25.419848 30599 hybrid_clock.cc:648] HybridClock initialized: now 1786545145419833 us; error 30 us; skew 500 ppm
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:32:25.422396 30599 init.cc:377] Logged in from keytab as kudu/127.23.173.252@KRBTEST.COM (short username kudu)
I20260812 14:32:25.422873 30599 webserver.cc:460] Webserver started at http://127.23.173.252:42671/ using document root <none> and password file <none>
I20260812 14:32:25.423098 30599 fs_manager.cc:359] Metadata directory not provided
I20260812 14:32:25.423148 30599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:32:25.423280 30599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:32:25.424383 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data/instance:
uuid: "84b6949d7df344ce966348db1e2f8437"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "290a739b407dea58bd679b1377903c5c"
server_key_iv: "5122ad6f02a3e6bb3278bd14493d3df2"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.424736 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal/instance:
uuid: "84b6949d7df344ce966348db1e2f8437"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "290a739b407dea58bd679b1377903c5c"
server_key_iv: "5122ad6f02a3e6bb3278bd14493d3df2"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.425979 30599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:32:25.426607 30624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:32:25.426757 30599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:32:25.426837 30599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data,/tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal
uuid: "84b6949d7df344ce966348db1e2f8437"
format_stamp: "Formatted at 2026-08-12 14:32:25 on dist-test-slave-1l3l"
server_key: "290a739b407dea58bd679b1377903c5c"
server_key_iv: "5122ad6f02a3e6bb3278bd14493d3df2"
server_key_version: "encryptionkey@0"
I20260812 14:32:25.426903 30599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:32:25.428068 30591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } attempt: 2
I20260812 14:32:25.428433 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.428308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:35996)
0812 14:32:25.428362 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.428400 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:32:25.428741 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.428656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:36006)
0812 14:32:25.428694 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.428721 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:32:25.428817 30591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.252:37441: Network error: Client connection negotiation failed: client connection to 127.23.173.252:37441: connect: Connection refused (error 111)
I20260812 14:32:25.446252 30599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:32:25.448400 30599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:32:25.448524 30599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:32:25.452394 30599 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.252:37441
I20260812 14:32:25.452440 30712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.252:37441 every 8 connection(s)
I20260812 14:32:25.452768 30599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/data/info.pb
I20260812 14:32:25.453645 30713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:32:25.455157 30713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:25.461018 24247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhwVqn/build/release/bin/kudu as pid 30599
I20260812 14:32:25.461125 24247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhwVqn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544840008271-24247-0/minicluster-data/master-2/wal/instance
I20260812 14:32:25.461345 24247 external_mini_cluster.cc:1428] Setting key 032059b16a57c072974db1395dba1676
I20260812 14:32:25.463791 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.455841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.23.173.254:37421)
0812 14:32:25.455896 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.455898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.456075 (+ 177us) server_negotiation.cc:407] Connection header received
0812 14:32:25.456568 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.456568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.456577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.456635 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.457191 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.457652 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.458587 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.458764 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.458807 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.460945 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.460946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.460947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.460954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.462505 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.462890 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.462891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.462892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.462946 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.463190 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.463190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.463190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.463291 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.463338 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.463611 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.463673 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:32:25.463984 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.455811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.254:37421 (local address 127.0.0.1:39688)
0812 14:32:25.455960 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.456002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.456490 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.456687 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.456690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.456931 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.457124 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.457129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.457733 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.457734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.458520 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.458523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.458815 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.458816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.458915 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.459373 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.459381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.460842 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.462584 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.462588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.462589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.462818 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.463002 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.463003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.463003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.463131 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.463339 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.463343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.463493 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.463597 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.463767 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260812 14:32:25.464526 30713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:32:25.470355 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.461698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.23.173.252:37441)
0812 14:32:25.461871 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.461876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.461952 (+ 76us) server_negotiation.cc:407] Connection header received
0812 14:32:25.462019 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.462021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.462032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.462091 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.462805 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.463798 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.464730 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.465164 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.465223 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.467642 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.467653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.467653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.467660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.469103 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.469464 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.469467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.469468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.469522 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.469749 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.469750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.469751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.469937 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.470113 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.470180 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.470273 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260812 14:32:25.471158 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.464775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.253:42351 (local address 127.0.0.1:53714)
0812 14:32:25.464824 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.464835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.464873 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.465007 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.465008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.465115 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.465164 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.465166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.465771 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.465772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.466545 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.466548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.466873 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.466874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.466957 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.467348 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.467353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.468493 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.470091 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.470092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.470092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.470326 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.470539 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.470539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.470540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.470607 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.470875 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.470876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.470918 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.470979 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.471067 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:32:25.471163 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.464847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.23.173.253:42351)
0812 14:32:25.464899 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.464901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.464910 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:32:25.464935 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.464936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.464946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.464994 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.465234 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.465698 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.466618 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.466805 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.466859 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.468595 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.468597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.468597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.468605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.470013 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.470393 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.470395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.470395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.470467 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.470667 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.470668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.470668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.470800 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.470868 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.471012 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.471079 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:32:25.471560 30713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.252@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.254@KRBTEST.COM
I20260812 14:32:25.477548 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.471805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:36024)
0812 14:32:25.471841 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.471849 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.471885 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.471981 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.471982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.472101 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.472124 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.472125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.472678 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.472679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.473124 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.473125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.473445 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.473446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.473493 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.473832 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.473836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.474993 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.476578 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.476579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.476579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.476779 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.476961 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.476962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.476962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.477006 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.477231 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.477232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.477277 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.477349 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.477442 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:32:25.477617 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.471861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.23.173.252:37441)
0812 14:32:25.471895 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.471896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.471905 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:32:25.471928 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.471929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.471937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.471980 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.472184 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.472616 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.473186 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.473386 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.473438 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.475059 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.475060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.475060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.475067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.476492 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.476831 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.476832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.476833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.476889 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.477058 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.477059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.477059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.477175 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.477235 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.477370 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.477497 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":19}
I20260812 14:32:25.478456 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.471274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.23.173.254:37421)
0812 14:32:25.471315 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.471316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.471323 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:32:25.471345 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.471346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.471355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.471409 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.471940 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.472384 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.473022 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.473218 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.473271 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.475654 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.475655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.475656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.475663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.477137 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.477493 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.477495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.477495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.477552 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.477776 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.477778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.477778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.477911 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.478072 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.478295 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.478372 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:32:25.479301 30713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: Bootstrap starting.
I20260812 14:32:25.479846 30480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } attempt: 2
I20260812 14:32:25.480474 30713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:25.480810 30713 log.cc:824] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:25.481664 30713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: No bootstrap required, opened a new log
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.253@KRBTEST.COM
I20260812 14:32:25.483553 30713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.483686 30713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b6949d7df344ce966348db1e2f8437, State: Initialized, Role: FOLLOWER
I20260812 14:32:25.483801 30713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.484140 30713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:25.484107 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.484201 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.254@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:32:25.486666 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.479065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.23.173.253:42351)
0812 14:32:25.479102 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.479104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.479117 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:32:25.479180 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.479180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.479189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.479230 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.479783 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.480382 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.481112 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.481324 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.481391 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.483800 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.483801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.483802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.483810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.485462 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.485805 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.485806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.485807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.485867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.486154 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.486155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.486156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.486296 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.486469 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.486506 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.486579 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:32:25.486545 30739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:25.487180 30739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:25.487353 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.480105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:36038)
0812 14:32:25.480149 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.480158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.480189 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.480299 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.480299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.480444 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.480488 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.480490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.481302 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.481303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.482014 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.482017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.482498 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.482499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.482558 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.483019 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.483024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.484681 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.486256 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.486257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.486258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.486485 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.486700 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.486701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.486701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.486761 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.487037 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.487039 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.487090 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.487157 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.487258 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:32:25.487746 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.480168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.23.173.252:37441)
0812 14:32:25.480203 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.480204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.480211 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:32:25.480230 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.480231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.480239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.480293 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.480574 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.481202 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.482113 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.482414 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.482481 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.484769 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.484771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.484771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.484778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.486182 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.486571 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.486572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.486573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.486630 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.486822 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.486822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.486823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.486958 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.487024 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.487554 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.487636 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:32:25.488891 24247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:32:25.489609 30480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a: Bootstrap starting.
I20260812 14:32:25.489637 30436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "737ef3931b564de485023564c3b5015a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704
I20260812 14:32:25.490346 30480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:25.487771 30599 master_runner.cc:186] Error getting master registration for 127.23.173.254:37421: OK, instance_id { permanent_uuid: "737ef3931b564de485023564c3b5015a" instance_seqno: 1786545145138201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:32:25.490562 30599 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:32:25.490651 30480 log.cc:824] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:25.491466 30480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a: No bootstrap required, opened a new log
I20260812 14:32:25.492828 30480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.492964 30480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737ef3931b564de485023564c3b5015a, State: Initialized, Role: FOLLOWER
I20260812 14:32:25.493054 30480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.493381 30480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:25.493346 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.493448 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:25.493501 30436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 14:32:25.493567 30436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:25.494381 30436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.494629 30436 leader_election.cc:288] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [CANDIDATE]: Term 1 election: Requested vote from peers 7c85c2ca9f5b48b897723d4ff26f4446 (127.23.173.253:42351), 84b6949d7df344ce966348db1e2f8437 (127.23.173.252:37441)
I20260812 14:32:25.494997 30668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737ef3931b564de485023564c3b5015a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84b6949d7df344ce966348db1e2f8437"
I20260812 14:32:25.495121 30668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:25.496148 30668 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737ef3931b564de485023564c3b5015a in term 1.
I20260812 14:32:25.496423 30391 leader_election.cc:302] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737ef3931b564de485023564c3b5015a, 84b6949d7df344ce966348db1e2f8437; no voters:
I20260812 14:32:25.496632 30743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:32:25.496784 30743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [term 1 LEADER]: Becoming Leader. State: Replica: 737ef3931b564de485023564c3b5015a, State: Running, Role: LEADER
I20260812 14:32:25.496872 30743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.497346 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737ef3931b564de485023564c3b5015a. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.497423 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:25.498284 30359 master_runner.cc:186] Error getting master registration for 127.23.173.253:42351: OK, instance_id { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" instance_seqno: 1786545145287502 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:32:25.501137 30359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 14:32:25.501631 30761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:25.501683 30761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:25.501725 30774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:32:25.501878 30774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:32:25.503904 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.495989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.23.173.253:42351)
0812 14:32:25.496039 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.496041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.496278 (+ 237us) server_negotiation.cc:407] Connection header received
0812 14:32:25.496317 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.496318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.496329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.496379 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.497149 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.497874 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.498962 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.499253 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.499325 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.500879 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.500881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.500881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.500889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.502584 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.502981 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.502983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.502983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.503047 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.503264 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.503265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.503265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.503405 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.503479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.503732 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.503815 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:32:25.504315 30668 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [term 1 FOLLOWER]: Refusing update from remote peer 737ef3931b564de485023564c3b5015a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:32:25.505012 30772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:32:25.506425 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737ef3931b564de485023564c3b5015a. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.506536 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:25.511658 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.502843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.23.173.254:37421)
0812 14:32:25.502885 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.502887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.504607 (+ 1720us) server_negotiation.cc:407] Connection header received
0812 14:32:25.504711 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.504712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.504723 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.504769 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.505444 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.506103 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.507099 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.507396 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.507453 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.508867 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.508869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.508869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.508876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.510440 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.510848 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.510849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.510850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.510908 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.511087 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.511088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.511089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.511239 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.511319 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.511492 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.511571 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:32:25.518604 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.503246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.23.173.252:37441)
0812 14:32:25.503295 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.503297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.511674 (+ 8377us) server_negotiation.cc:407] Connection header received
0812 14:32:25.511890 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.511891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.511903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.511932 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.512550 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.513007 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.513948 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.514256 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.514314 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.515660 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.515662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.515662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.515669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.517329 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.517771 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.517772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.517773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.517831 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.517999 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.518000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.518000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.518166 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.518252 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.518426 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.518508 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:32:25.519939 30591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } attempt: 3
I20260812 14:32:25.521284 30772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.521409 30772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:25.522001 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.522104 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:25.524796 30776 mvcc.cc:205] Tried to move back new op lower bound from 7317688915983179776 to 7317688915956039680. Current Snapshot: MvccSnapshot[applied={T|T < 7317688915983179776}]
Aug 12 14:32:25 dist-test-slave-1l3l krb5kdc[30339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.253@KRBTEST.COM for kudu/127.23.173.252@KRBTEST.COM
I20260812 14:32:25.526520 30774 catalog_manager.cc:1347] Generated new cluster ID: 210bc43a50b44443b85aa0fadd45db90
I20260812 14:32:25.526580 30774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:32:25.530375 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.520204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.252:37441 (local address 127.0.0.1:36056)
0812 14:32:25.520271 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 14:32:25.520282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:32:25.520329 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:32:25.520454 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:32:25.520455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:32:25.520628 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:32:25.520678 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.520680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.521694 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.521695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.522486 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:32:25.522490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.523072 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.523073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:32:25.523147 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.523746 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:32:25.523752 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:32:25.525327 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:32:25.528945 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.528947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.528947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.529165 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.529459 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:32:25.529460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:32:25.529460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:32:25.529526 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:32:25.530027 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:32:25.530029 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:32:25.530094 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 14:32:25.530143 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:32:25.530258 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 14:32:25.531224 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:25.520245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.23.173.252:37441)
0812 14:32:25.520290 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.520292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.520313 (+ 21us) server_negotiation.cc:407] Connection header received
0812 14:32:25.520374 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.520375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.520384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.520441 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.520738 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.521368 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.522576 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:25.522817 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:25.522874 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:25.525427 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:32:25.525429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:32:25.525429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:32:25.525437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:32:25.528808 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.529252 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.529253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.529253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.529376 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:32:25.529618 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:32:25.529618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:32:25.529618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:32:25.529893 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:32:25.530840 (+ 947us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:25.530925 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:32:25.531058 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:32:25.533020 30591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: Bootstrap starting.
I20260812 14:32:25.533788 30591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: Neither blocks nor log segments found. Creating new log.
I20260812 14:32:25.534132 30591 log.cc:824] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: Log is configured to *not* fsync() on all Append() calls
I20260812 14:32:25.540506 30591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: No bootstrap required, opened a new log
I20260812 14:32:25.542980 30591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.543237 30591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c85c2ca9f5b48b897723d4ff26f4446, State: Initialized, Role: FOLLOWER
I20260812 14:32:25.543437 30591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } }
I20260812 14:32:25.543885 30774 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:32:25.544178 30549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:32:25.544610 30774 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:32:25.545099 30549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [term 1 FOLLOWER]: Refusing update from remote peer 737ef3931b564de485023564c3b5015a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:32:25.545377 30550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737ef3931b564de485023564c3b5015a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c85c2ca9f5b48b897723d4ff26f4446"
I20260812 14:32:25.545404 30772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:32:25.546147 30550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737ef3931b564de485023564c3b5015a in term 1.
I20260812 14:32:25.544080 30591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:32:25.548444 30792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.548563 30792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:25.548872 30792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737ef3931b564de485023564c3b5015a. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.549330 30792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:25.549391 30792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.549441 30792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:32:25.549705 30774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a: Generated new TSK 0
I20260812 14:32:25.549877 30774 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:32:25.551441 30817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:32:25.551497 30817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:32:25.552495 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737ef3931b564de485023564c3b5015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737ef3931b564de485023564c3b5015a" member_type: VOTER last_known_addr { host: "127.23.173.254" port: 37421 } } peers { permanent_uuid: "7c85c2ca9f5b48b897723d4ff26f4446" member_type: VOTER last_known_addr { host: "127.23.173.253" port: 42351 } } peers { permanent_uuid: "84b6949d7df344ce966348db1e2f8437" member_type: VOTER last_known_addr { host: "127.23.173.252" port: 37441 } } }
I20260812 14:32:25.552592 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737ef3931b564de485023564c3b5015a [sys.catalog]: This master's current role is: LEADER
I20260812 14:32:25.560316 30804 mvcc.cc:205] Tried to move back new op lower bound from 7317688916141215744 to 7317688915956039680. Current Snapshot: MvccSnapshot[applied={T|T < 7317688915983179776}]
W20260812 14:32:25.577522 30486 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.575791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.23.173.254:37421)
0812 14:32:25.575854 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.575857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.576822 (+ 965us) server_negotiation.cc:407] Connection header received
0812 14:32:25.576930 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.576931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.576943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.576974 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.577448 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39734: BlockingRecv error: recv got EOF from 127.0.0.1:39734 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 14:32:25.578311 30719 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.576187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.23.173.252:37441)
0812 14:32:25.576228 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.576230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.577778 (+ 1548us) server_negotiation.cc:407] Connection header received
0812 14:32:25.577876 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.577877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.577889 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.577917 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.578243 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36058: BlockingRecv error: recv got EOF from 127.0.0.1:36058 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:32:25.579058 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.576528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.23.173.253:42351)
0812 14:32:25.576569 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.576571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.578540 (+ 1969us) server_negotiation.cc:407] Connection header received
0812 14:32:25.578634 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.578635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.578645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.578674 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.578982 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53752: BlockingRecv error: recv got EOF from 127.0.0.1:53752 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 14:32:25.581419 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.580699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.23.173.253:42351)
0812 14:32:25.580734 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.580736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.580911 (+ 175us) server_negotiation.cc:407] Connection header received
0812 14:32:25.581011 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.581012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.581043 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.581071 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.581374 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53764: BlockingRecv error: recv got EOF from 127.0.0.1:53764 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:32:25.584456 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.583657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.23.173.253:42351)
0812 14:32:25.583688 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.583690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.583985 (+ 295us) server_negotiation.cc:407] Connection header received
0812 14:32:25.584083 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.584084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.584094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.584121 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.584414 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53780: BlockingRecv error: recv got EOF from 127.0.0.1:53780 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:32:25.588320 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.587578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.23.173.253:42351)
0812 14:32:25.587609 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.587610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.587842 (+ 232us) server_negotiation.cc:407] Connection header received
0812 14:32:25.587928 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.587928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.587938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.587966 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.588276 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53788: BlockingRecv error: recv got EOF from 127.0.0.1:53788 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 14:32:25.594245 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.593456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.23.173.253:42351)
0812 14:32:25.593489 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.593490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.593757 (+ 267us) server_negotiation.cc:407] Connection header received
0812 14:32:25.593853 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.593854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.593864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.593889 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.594201 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53802: BlockingRecv error: recv got EOF from 127.0.0.1:53802 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:32:25.604228 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.603434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.23.173.253:42351)
0812 14:32:25.603468 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.603469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.603686 (+ 217us) server_negotiation.cc:407] Connection header received
0812 14:32:25.603803 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.603804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.603814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.603841 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.604170 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53808: BlockingRecv error: recv got EOF from 127.0.0.1:53808 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:32:25.624023 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.621584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.23.173.253:42351)
0812 14:32:25.621625 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.621628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.623356 (+ 1728us) server_negotiation.cc:407] Connection header received
0812 14:32:25.623470 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.623471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.623485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.623516 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.623964 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53814: BlockingRecv error: recv got EOF from 127.0.0.1:53814 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:32:25.658356 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.657448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.23.173.253:42351)
0812 14:32:25.657497 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.657499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.657756 (+ 257us) server_negotiation.cc:407] Connection header received
0812 14:32:25.657850 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.657852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.657863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.657891 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.658293 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53824: BlockingRecv error: recv got EOF from 127.0.0.1:53824 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 14:32:25.724371 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.723744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.23.173.253:42351)
0812 14:32:25.723834 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.723836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.723884 (+ 48us) server_negotiation.cc:407] Connection header received
0812 14:32:25.723943 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.723944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.723954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.724006 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.724307 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53840: BlockingRecv error: recv got EOF from 127.0.0.1:53840 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 14:32:25.854315 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:25.853637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.23.173.253:42351)
0812 14:32:25.853740 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:25.853742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:25.853759 (+ 17us) server_negotiation.cc:407] Connection header received
0812 14:32:25.853835 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:25.853836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:25.853846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:25.853909 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:25.854248 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53854: BlockingRecv error: recv got EOF from 127.0.0.1:53854 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 14:32:26.112056 30596 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:32:26.111501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.23.173.253:42351)
0812 14:32:26.111558 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:26.111561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:26.111571 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:32:26.111601 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:26.111602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:26.111613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:26.111663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:32:26.111990 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53868: BlockingRecv error: recv got EOF from 127.0.0.1:53868 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:32:26.487669 30739 catalog_manager.cc:1259] Loaded cluster ID: 210bc43a50b44443b85aa0fadd45db90
I20260812 14:32:26.487778 30739 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: loading cluster ID for follower catalog manager: success
I20260812 14:32:26.488629 30739 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: acquiring CA information for follower catalog manager: success
I20260812 14:32:26.488940 30739 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84b6949d7df344ce966348db1e2f8437: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:32:26.551940 30817 catalog_manager.cc:1259] Loaded cluster ID: 210bc43a50b44443b85aa0fadd45db90
I20260812 14:32:26.552016 30817 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: loading cluster ID for follower catalog manager: success
I20260812 14:32:26.552848 30817 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: acquiring CA information for follower catalog manager: success
I20260812 14:32:26.553166 30817 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c85c2ca9f5b48b897723d4ff26f4446: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:32:26.627687 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:26.625205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.23.173.253:42351)
0812 14:32:26.625317 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:26.625319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:26.625357 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:32:26.625430 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:26.625431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:26.625483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:26.625565 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:26.625687 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.626198 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.626976 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.627185 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.627242 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:26.627394 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:26.627506 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:26.627572 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:26.627588 (+ 16us) server_negotiation.cc:299] Negotiation successful
0812 14:32:26.627606 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260812 14:32:26.631731 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:26.629381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.23.173.252:37441)
0812 14:32:26.629523 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:26.629526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:26.629537 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:32:26.629584 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:26.629585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:26.629626 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:26.629694 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:26.629802 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.630283 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.631050 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.631253 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.631314 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:26.631453 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:26.631544 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:26.631595 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:26.631612 (+ 17us) server_negotiation.cc:299] Negotiation successful
0812 14:32:26.631629 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260812 14:32:26.636382 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:26.633386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.23.173.253:42351)
0812 14:32:26.633436 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:26.633438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:26.633527 (+ 89us) server_negotiation.cc:407] Connection header received
0812 14:32:26.633751 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:26.633752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:26.633761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:26.633799 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:26.633935 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.634635 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.635601 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.635918 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.635983 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:26.636146 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:26.636178 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:26.636204 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:26.636293 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:32:26.636317 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:32:26.636760 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:26.633592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.23.173.252:37441)
0812 14:32:26.633645 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:26.633647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:26.633653 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:32:26.633672 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:26.633672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:26.633679 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:26.633716 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:26.633830 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.634545 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.636021 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.636281 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.636332 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:26.636516 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:26.636546 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:26.636572 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:26.636678 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 14:32:26.636699 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:32:26.637131 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:32:26.633291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.23.173.254:37421)
0812 14:32:26.633390 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 14:32:26.633392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:32:26.633400 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:32:26.633456 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:32:26.633457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:32:26.633509 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:32:26.633626 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:32:26.634719 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.635194 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.636130 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:32:26.636427 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:32:26.636627 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:32:26.636701 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:32:26.636797 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:32:26.637011 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 14:32:26.637029 (+ 18us) server_negotiation.cc:299] Negotiation successful
0812 14:32:26.637054 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20260812 14:32:26.638810 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30359
I20260812 14:32:26.650488 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30489
I20260812 14:32:26.661417 24247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOhwVqn/build/release/bin/kudu with pid 30599
2026-08-12T14:32:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2963 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5903 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (306661 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS